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Executive Summary 


The Literacy Design Collaborative (LDC) was created to support teachers in 
implementing College and Career Readiness Standards in order to teach literacy skills 
throughout the content areas. The LDC Investing in Innovation (i3) project focuses on 
developing teacher competencies through job-embedded professional development and the 
use of professional learning communities (PLCs). Teachers work collaboratively with coaches 
to further develop their expertise and design standards-driven, literacy-rich writing 


assignments within their existing curriculum across all content areas. 


Engaged in the evaluation of LDC tools since June 2011, UCLA’s National Center for 
Research on Evaluation, Standards, and Student Testing (CRESST) is the independent 
evaluator for LDC’s federally funded Investing in Innovation (i3) validation grant. The 2016- 
17 school year was the first year of implementation, following a pilot year during which the 
implementation plan, instruments, data collection processes, and analytical methodologies 


were refined. 


This annual report presents an initial look at LDC implementation in the first cohort of 
20 schools in a large West Coast district during their first year of implementation. The early 
results suggest the following: 


e Participants across all groups reported positive attitudes toward LDC. All measures 
of satisfaction or improvement were rated positively by more than half of 
respondents. Two thirds of teachers expressed interest in learning more about how 
to lead LDC implementation at their schools, and over half of project liaisons and 
administrators anticipated that their teachers would continue with LDC the following 


year. 


e Participants perceive a positive impact on student outcomes. Three quarters of 
teachers and 95% of administrators agreed that LDC helped improve students’ 
literacy performance. In particular, teachers reported high impact on writing quality, 
college and career readiness skills, overall literacy performance, reading skills, and 


content knowledge. 


e Individuals leading and supporting the LDC implementation at all levels received 
highly positive ratings. LDC coaches were rated by 95% of teachers as providing 
appropriate and timely feedback. Project liaisons were almost universally reported 
to be highly approachable, effective, and knowledgeable. Almost all teachers 
reported that their administrators encouraged LDC participation in schools. A large 


5majority of project liaisons and administrators had positive interactions with LDC 


staff and were able to receive appropriate resources and support when needed. 


e Analysis of module artifacts suggest that teachers at the elementary school level 
were moderately successful in the backwards design process, particularly in 
developing high quality writing tasks for students. This was evidenced in the mean 
ratings that were generally in the three (moderately present or realized) range both 


for the overall elementary sample and content area subgroups. 


e At this point, there is insufficient quantitative evidence to suggest a positive LDC 
impact on student test scores either at the elementary or middle school level. This 
finding should not be surprising given the early stage of intervention, with teachers 


having only completed one year of the two-year implementation process. 


e The LDC intervention appears to have differential results for teachers in different 
content areas. It seems to be a better fit for English language arts and history/social 
studies teachers than for science and math teachers. Teacher feedback, module 
scores, and level of engagement with CoreTools all indicated that science and math 


teachers were less engaged with the material and experienced less success. 


e This district’s implementation did not, on average, appear to have met LDC’s 
participation expectations for high implementation. The ideal is that PLC members 
meet weekly for at least 60 minutes. Only 30% of teachers reported meeting at least 
once a week or more. Almost half (46%) met every other week. Almost three 
quarters reported that meetings lasted 45 minutes to an hour, and a quarter reported 
they lasted longer than an hour. That said, 70% of teachers agreed that their PLC 
was given sufficient time to meet, although many teachers who provided open-ended 


responses asked for more protected, paid time. 


As an ongoing multi-year intervention, the LDC implementation will continue to evolve 
year to year as participants provide feedback and LDC program managers make refinements. 
Thus, we anticipate that further significant changes to the course material and the delivery 
system that are already in progress for Year 2 will likely result in continued and possibly 
increased positive feedback. Related, we posit that further support for science and math 
teachers would likely result in higher levels of success and satisfaction for those teachers. 
Finally, as teachers return for a second year and achieve greater experience with the LDC 
model, it is likely that their ability to apply their learning in increasingly productive ways will 
become more evident in their self-reports, module quality, and engagement with the LDC 
platform. 


Literacy Design Collaborative: 2016-17 Evaluation Report 


Jia Wang, Joan Herman, Scott Epstein, Seth Leon, Deborah La Torre, Julie 
Haubner, and Velette Bozeman 


CRESST/University of California, Los Angeles 


1.0 Introduction 


The Literacy Design Collaborative (LDC) was created to support teachers in 
implementing College and Career Readiness Standards in order to teach literacy skills 
throughout the content areas. The LDC Investing in Innovation (i3) project focuses on 
developing teacher competencies through job-embedded professional development and the 
use of professional learning communities (PLCs). Teachers work collaboratively with coaches 
to further develop their expertise and design standards-driven, literacy-rich writing 
assignments within their existing curriculum across all content areas. LDC is a national 
community of educators providing a teacher-designed and research-based framework, online 
tools, and resources for creating both literacy-rich assignments and courses across content 
areas. Used by individual teachers, schools, and districts in 40 states for the past four years, 
LDC also is a statewide adopted strategy for Common Core implementation in Kentucky, 
Colorado, Louisiana, and Georgia. 


UCLA’s National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST), in collaboration with its partner Research for Action (RFA), engaged in the 
evaluation of the implementation and impact of LDC tools on student learning and teacher 
effectiveness starting in June 2011, via two parallel research studies funded by the Bill and 
Melinda Gates Foundation. Those studies included an examination of LDC implementation in 
eighth grade social studies and science classrooms in Kentucky and Pennsylvania and a 
district-wide implementation in sixth grade advanced reading classes in a large district in 
Florida. Results for the studies are available in two technical reports (Herman et al., 2015a; 
Herman et al., 2015b), as well as a journal article published by AERA Open (Herman, Epstein 
& Leon, 2016). 


Currently, CRESST serves as the independent evaluator for LDC’s federally funded 
Investing in Innovation (i3) validation grant. The LDC i3 study is examining the 
implementation and impact of LDC in two large school districts: New York City Department of 
Education and a large school district on the West Coast. The evaluation study is a 


comprehensive mixed-method evaluation to understand the impact of LDC on student 


learning using a quasi-experimental design, as well as to document impact on teacher skills 
and practices. Specifically, the evaluation study addresses a rich range of questions about 
program characteristics, conditions and program impacts in the context of two large urban 
school districts. The study will draw on data from two cohorts of schools, with each school 
housing a professional learning community (PLC) of teachers engaging in professional learning 
about LDC and implementing LDC mini-tasks and modules in their classrooms. We will 
measure teacher implementation and skill improvement via teacher surveys, analysis of 
analytic data from LDC’s online CoreTools module building platform, and artifact analysis. 
While we will document the core strategies of the LDC model as implemented and provide 
support for LDC improvement, the central focus of our comprehensive mixed-method 
evaluation is examining the impact of LDC on teacher practices and student learning using a 


quasi-experimental design. 


The first i3 evaluation cohort of schools began implementing LDC during the 2016-17 
school year. This annual progress report examines LDC implementation during the 2016-17 
school year in a large school district on the West Coast, and presents the first exploratory 
analyses of the impact of LDC on student learning in evaluation cohort schools. A parallel 
progress report focuses on implementation in the New York City Department of Education 
(NYCDOE). The current annual progress report presents results from (a) analyses describing 
how LDC participants interacted with the CoreTools module building platform; (b) scoring by 
CRESST of instructional modules created by LDC participants; (c) surveys of classroom 
teachers, LDC project liaisons, and school administrators; and (d) student outcome analyses 


using the quasi-experimental design. 


These results provide a window into how LDC was implemented in 2016-17, the 
perceived utility and effectiveness of various program components, and the perceived impact 
of LDC on both teacher and student skills and knowledge. A preliminary test of the 


effectiveness of LDC in increasing student learning is also included in the report. 


1.1 Logic Model 


The logic model includes four key intervention components that were predicted to be 
the drivers of change in teacher practice and student learning (see Figure 1.1). These are a 
coach-supported Professional Learning Community formed to implement the LDC 
intervention at the school site and provide a space for teacher collaboration; asynchronous 
support from coaches in the form of feedback in CoreTools through comments and peer 


review; implementation activities completed by participating teachers including module 


development and classroom implementation; and leadership support at different levels. 
Note that the model also indicates LDC’s implementation expectations in each area. 


LITERACY DESIGN COLLABORATIVE i3 LOGIC MODEL 


Activities Participation Short term Medium term Long term 


Common Planning Student Increased Students 

timetor LDC PLC members participate in at least 60 engagement CCRS skills acquire 

Professional minutes of common planning time during knowledge and 
each consecutive two weeks. Improved skills to be 


Learning 
Community (PLC) 
with synchronous 
coach support 


Digital coach joins PLC every other week. test scores college and 
Sequencing incorporates blocks from LDC 
online courses. 


career ready 


Higher rates 

of course Educational 
completion attainment 
increases 


Increased teacher 


expertise and skill 
development 


Asynchronous 
support from LDC 


Coach provides feedback on modules in 
CoreTools via comments and peer review Higher 


rubric ratings. graduation 
rates 


coaches 


Increased labor 


market 
Teacher - Teachers develop at least one module per outcomes 
implementation year in Years 1-2. 
activities - Teachers teach at least two modules per year 
in Years 1-3. 
More effective 
CCRS-aligned 
instruction 
District instructional specialists visit schools and use of 
three times per year to sit in on PLC, meet with formative 
Leadership principal and teacher leader, and observe assessment 
support at instruction. 


different levels - Principals participate in LDC by protecting PLC 
time, setting goals for LDC work, facilitating & 
participating in school visits from district 
instructional specialists, and observing LDC 
instruction. 
Teacher leaders facilitate PLC on the ground, 
collaborate with administrator, and 
communicate regularly with coach. 


Contextual Factors: District support, school and student demographics, other policies and initiatives 


Figure 1.1. LDC i3 Logic Model. 


The logic model predicts that the four key components will lead to increased teacher 
expertise and skill development and more effective Common Core aligned instruction which 
incorporates formative assessment. In turn, increased teacher capacity and more effective 
instruction will lead to increased student engagement in the short term; increased student 
skill acquisition, higher test scores, and higher rates of course completion in the medium 
term; and improved college and career readiness, education attainment, graduation rates, 


and labor market outcomes in the long term. 


Note that the logic model has been revised based on refinements to the program in 
response to learning from the pilot year (2015-16) and the first year for implementation 
cohort 1 (2016-17). The logic model presented here is current as of Fall 2017. 


Note also that Figure 1.1 refers to teacher leaders, but this report will refer to project 
liaisons. That distinction reflects an update to the model; starting in 2017-18, teacher leaders 


will be identified in the first year that a school implements LDC, and those teacher leaders 
will receive a stipend in their first year. This change was not yet in effect during the 2016-17 
school year, and we therefore refer to teachers playing a leadership role in LDC as project 


liaisons. 


1.2 Evaluation Questions 


Our evaluation questions focus on addressing three main areas: program characteristics 
and implementation, contextual factors and implementation, and program impacts. This 
progress report provides findings on many, but not all, of the evaluation questions. In 
particular, given that the evaluation is still in its early stages, there is limited information 
available regarding program impacts. This report provides a first look at how the refined LDC 
model is impacting student learning, although the quasi-experimental design analyses 
contained herein should be considered exploratory rather than confirmatory. The first 
confirmatory analysis will be conducted at the end of two years of participation for the 


teachers. 


I. Program Characteristics and Implementation 


a. Who are the participating teachers and schools? Are they representative of the 
teacher/school populations of the respective district on years of teaching, 
education level, prior student performance, etc.? 


b. How is the LDC program implemented in each district? What are the core 
components (e.g., training, tools, on-site or other direct support) and who are 
the key participants? In what ways did the LDC implementation align with the 
intended model? 


c. In what ways do teachers implement the LDC tools in their classrooms? To what 
extent do teacher practices align with intended LDC practices? 


d. How are teachers utilizing the online LDC system (including online tools, 
exemplars, collaborative work spaces, and technical assistance) in terms of 
frequency and use of key features? Does this vary by teacher characteristics? 
What are teachers’ perceptions of the value and quality of the online LDC system? 


e. What types of LDC professional development opportunities are offered to and 
utilized by teachers at each school/district? Are teachers and schools satisfied 
with the LDC professional development opportunities they received? 


II. Contextual Factors and Implementation 


a. What factors facilitate or hinder successful implementation of the LDC model at 
the teacher, school, and district levels? 


b. How can implementation of the model be improved at the teacher, school, and 
district levels? 


Til. 


a. 


What other educational reforms are being implemented in the participating 
schools and districts? What are their influences on the LDC adoption in the 
schools and districts? Are schools able to align reform efforts? 


What are the roles of school and district leadership in shaping the LDC 
implementation? 


Program Impacts 


What is the impact of LDC on the academic performance of participating students 
as measured by the state assessments? 


Do the academic impacts vary by student subgroup including prior achievement, 
race, ethnicity, socio-economic status, gender, language proficiency, and/or 
disability? Does LDC help close the achievement gap between student subgroups? 


Do the academic impacts vary by student grade level or subject? 


What is the impact of LDC on teacher skill improvement and learning as measured 
by CoreTools and by the quality of LDC modules they produce? What is the self- 
reported impact of LDC on teacher learning? 


To what extent do teachers report changes in their practice (e.g., teaching 
strategy, collaboration with others) and changes in their comfort in implementing 
CCSS during and after the LDC intervention? 


What is the relationship between the fidelity of implementation, fidelity of 
intervention, and student learning? What are the conditions and contexts under 
which the LDC tool use is most effective? 


To what extent do Cohort 1 participating schools and teachers continue their 
LDC-influenced practices in the 2019-20 school year after the LDC support ends? 
What contributed to their decision to continue or stop? What factors contributed 
to their levels of continued implementation? How does Cohort 1’s actions align 
with their previously stated intentions for continuation of LDC-influenced 
practices as reported in spring 2017? To what extent do Cohort 2 participating 
schools and teachers plan to continue their LDC-influenced practices after the 
LDC support ends? 
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2.0 Study Methodology 


In this chapter we provide an overview of the methodology behind this early look at LDC 
in 2016-17. We begin by describing the various instruments and data sources for the analyses, 
including (a) analytic data from LDC’s CoreTools platform; (b) module artifacts including 
samples of student work; (c) surveys of classroom teachers and project liaisons participating 
in PLCs and administrators overseeing the implementation; and (d) administrative data on 
students and teachers used for outcomes analyses. We then describe the sample of educators 
and schools for each of these data sources. Finally, we discuss the methodological approaches 


for the various analyses we conducted. 


2.1 Data and Instruments 


We describe below each of the data instruments and the elements they contain. Most 
variables are measured at the teacher-level, which is the unit at which the LDC intervention 
is being implemented. Administrative data for the analysis of the impact of LDC on student 


learning include school-, teacher-, and student-level variables. 


LDC CoreTools. The CRESST team received the LDC program data on i3 participants’ 
interactions with the CoreTools module building platform. The data files captured three key 
activities related to the module building platform: document page viewing, document editing, 


and document commenting. 


Specifically, the data contained date- and time-stamped records of participants’ 
activities in each of these areas, and we analyzed variation in the number of times the 
participants performed these activities across the school year. We generated descriptive 
statistics (minimum, maximum, mean, standard deviation) for the number of times 
participants viewed a document page, edited a module document, and commented on a 
module document. We also produced descriptive statistics on these behaviors for various role 
(teacher, project liaison, administrator), school level (elementary, middle, high) and content 
area subgroups. Finally, we examined the difference in average engagement in these key 
activities between teachers whose completed modules we rated in Chapter 4 of this report 
and those teachers who did not complete a module. 


Modules. Our existing rubrics, developed for our prior LDC evaluation work (Herman 
et al., 2015a), were adapted to examine the quality and coherence of the LDC modules and 
to address the quality of both content and literacy development materials (i.e., template task, 
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student work samples, and descriptions of the pacing and goals of the modules). The six 
dimensions examined for this study included the following: (1) effective writing task; (2) 
alignment to the CCSS and local and state literacy and content standards; (3) fidelity to LDC 
module instruction; (4) quality of instructional strategies; (5) coherence and clarity of 
module; and, (6) overall impression. Three additional dimensions that focused on issues of 
text quality were excluded since submissions did not include copies of the materials used by 
the teachers. Each dimension was rated using a five-point scale with anchor points for the 
first five dimensions ranging from “not present or not realized” to “fully present or fully 
realized” and anchor points for the final dimension ranging from inadequate to advanced LDC 
module implementation. Detailed definitions of each dimension and descriptions for what 
constitutes ratings of 1, 3, and 5 on each dimension can be found in the rubric in Appendix 
A. 


Surveys (Teachers, Project Liaisons, and Administrators). In collaboration with 
LDC, CRESST made substantial revisions to pilot year (2015-16) surveys. Revisions address 
lessons learned from administration and analysis of the pilot surveys, adjustments to the 
program model made during and subsequent to the pilot year, and a desire to yield more 
robust information on teacher skills and practices. Items were also added to help understand 
in which grades and classes teachers were implementing LDC and to help identify the modules 
teachers were spending their time and energy on. Similar to the pilot year, CRESST designed 
five surveys to capture data on the experience of LDC participants playing three different 
roles: teacher, project liaison, and administrator. Some project liaisons were also 
administrators or teachers. Thus, five versions of the surveys were administered in spring 
2017: (1) teacher, (2) teacher/project liaison, (3) project liaison, (4) administrator, and (5) 


administrator/project liaison. 


The surveys were designed to capture multiple perspectives on key aspects of LDC’s 
logic model? (see Figure 1.1), and to provide data to answer the evaluation’s research 
questions presented earlier. Survey questions targeted at the three roles fall under the 
domains and sub-domains in Table 2.1. Domains were selected to align with the LDC i3 logic 
model and with the CRESST evaluation’s research questions. Note that most domains cover 


multiple sub-domains, constructs and survey’ questions. Professional Learning 


1 See Reisman, Herman, Luskin, and Epstein (2013) for a summary of the original 
generalizability study conducted using the CRESST developed rubrics. We excluded three 
dimensions that focused on issues of text quality as texts selected by teachers were not 
readily available in CoreTools for the analysis. 

2 The survey domains were aligned to this version of the Logic Model for the pilot year. The 
Logic Model has since been revised to align with the revised LDC implementation plan. 
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Community/Teacher Collaboration, for example, captures the intensity, frequency, and 
collaborative environment of common planning time; LDC Training and Support includes 
quality of online courses, utility and effectiveness of coach support, etc.; and LDC 
Implementation encompasses module creation, classroom implementation of modules, and 


module peer review. 


Table 2.1 
Survey Domains for Three Respondent Groups 
Domain Teacher ie Administrator 

LDC Participation X X X 
Professional Learning Community X X X 
and Teacher Collaboration 
LDC Training and Support Xx X X 
LDC Implementation 

Module Creation X X 

Classroom Implementation 

Module Peer Review X 

Alignment X X 
Leadership Support 

Project Liaison Support X 

School Administrator Support / 
Classroom Observation ‘. a a 

Project Liaison Leadership Role X 

District Support 
Impact 

Impact on Teacher Practice and X X 
Learning 

Impact on Student Learning X 
Scale-Up and Sustainability X X 
Facilitators and Barriers X 
Areas of Improvement X X X 


Teachers and administrators were asked to reflect on both LDC’s Impact on Teacher 
Practice and Learning and Impact on Student Learning. Questions within a number of domains 


further asked respondents to reflect on conditions and supports that may potentially impact 
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LDC’s implementation. These domains included teachers’ perception of Facilitators and 
Barriers to implementation and perceptions regarding leadership roles and support for LDC at 
different levels. Project liaisons and administrators were also asked for their perceptions 
regarding if and how LDC will be sustained and expanded within the school. Finally, all 
respondents were asked open-ended questions regarding Areas of Improvement for LDC 
implementation. Teacher, project liaison, and administrator surveys can be found in 
Appendices B, C, and D. 


Administrative Data used in Student Outcomes Analysis. Student-level variables 
utilized in the outcomes analysis included race/ethnicity, gender, poverty status, special 
education status, English language proficiency, gifted status, grade, and prior and current 
year achievement in math and ELA on state assessments. Teacher-level indicators obtained 
and utilized included years of teaching experience and teaching status (permanent, substitute, 
student teacher, etc.). We also requested and received roster files that establish a link 


between teachers and students via specific courses. 


2.2 Sample 


Twenty Cohort 1 schools began implementing the LDC program in the 2016-17 school 
year, with 154 classroom teachers participating and 34 administrators overseeing the work 
(see Table 2.2). The 20 schools included 11 elementary schools, four middle schools, and one 
high school, two K-8 schools, one 6-12 school, and one K-12 school. Participants taught across 
all grade levels from K to 12. Most secondary teachers taught ELA, social studies/history, or 
science, with a handful teaching other subjects such as math, foreign languages, special 


education, or the arts. 


As can be seen in Table 2.2, across the different measures data were available for a 
large majority of participants. Ninety-two percent of teachers consented to participate in the 
study, with 79 percent of all teachers completing the survey in spring 2017. The consent rate 
(82 percent) and survey response rate (75 percent) for administrators were a little lower than 
the corresponding rates for teachers. The CoreTools dataset, which was delivered to CRESST 
directly by LDC and did not depend on teachers’ individual study consents, captured a similar 
number of teachers and administrators to those that consented to the CRESST survey. 


In addition to the CoreTools analytic files, we also received module artifacts from LDC 
for an analysis of the quality of module design. We restricted our analysis to modules created 
during the 2016-17 school year that included original uploaded student work samples, 
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because these samples were required for module scoring. That restriction yielded a sample of 
53 modules that were authored or co-authored by 50 teachers (about a third of all 
participating teachers) and two administrators. Given the presence of uploaded student work, 
these are modules that we are confident were implemented in the classroom. It should be 
noted, however, that as described in Chapter 4, almost 80% of teachers made at least one 
edit to a mini-task or module in CoreTools. The 53 modules are therefore part of a larger 
universe of modules worked on by participating teachers; some of the modules which did not 


have uploaded student work may have also been implemented in classrooms. 


The school district required individually signed consent forms before releasing teacher 
data and teacher/student rosters, so for the outcomes analysis, we only received data on 
teachers who consented to participate in the study. The sample was further restricted by the 
need for student achievement data for both the outcome year (2016-17) and the prior year 
(2015-16). As a result, participants teaching either high school or lower elementary grades 
(K-3) could not be included in the student outcome analysis. These restrictions accounted for 
almost all of the reduction in the sample from 143 consenting teachers to 67 teachers in the 
outcomes analysis. Specifically, 23 high school teachers and 42 K-3 teachers were excluded. 
In addition, two middle school math teachers who consented to the study were excluded from 
the analysis as the rest of the middle school sample consisted of ELA, social studies/history, 
and science teachers; inclusion of the two math teachers would have complicated the 
structure of the models without substantially impacting the results. Six other consenting 
participants (e.g. literacy coach, instructional coach, RSP, etc.) were also excluded because 
they were not connected to specific classrooms. Finally, three additional teachers (two 
elementary school teachers and one middle school teacher) did not show up in the district 
administrative data, probably because they played roles such as special education that were 
not attached to classroom rosters. Further details on sample are presented in each of the 


results sections. 
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Table 2.2 


Number of Participating Schools, Teachers, and Administrators in the West Coast District 
and Sample Size for Different Instruments 


Number of Number of Number of 
classroom administrators schools in which 
teachers above teachers 
and 
administrators 
practice 
Participated in LDC 154 34 20 
Consented to CRESST study 142 28 20 
Completed survey 122 25 20 
Present in CoreTools dataset 141 30 12 
Author/co-author of module with 50 2 12 
student work uploaded 
Included in outcomes analysis 67 NA 19 


2.3 Module Scoring Process 


LDC requirements intended that all teachers implement two modules over the course of 
the year, with the first spanning one week based on one text and the second that spanned 
across multiple weeks and used multiple texts. Modules could be developed as original units 
of work or could be adapted from existing modules within the LDC CoreTools Library. Modules 
could also be either developed or adapted in solo or collaboratively with other teachers within 
the PLC. 


Modules included in this analysis were collected from elementary and secondary teachers 
who participated during the 2016-2017 school year. As noted earlier, all modules were 
retrieved from LDC’s CoreTools online platform and examined during summer 2017. Artifacts 
that included both a completed module template and at least one student work sample, as 
determined by the evaluation team and/or expert raters, were then included in the analysis. 
As a result, 53 modules were rated and analyzed for teachers who participated in the i3 study. 
Additional modules that were missing a significant number of components or that were later 


found ineligible for the study were dropped after the rating sessions. 


Expert raters with experience teaching in the targeted grade spans and content areas 
were recruited from schools in Los Angeles County. The panel included two elementary school 


teachers, two secondary science teachers, two secondary social studies teachers, and three 
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secondary English language arts (ELA) teachers. A greater number of raters was included for 


ELA since the number of modules eligible for analysis was greater for this content area. 


Separate trainings were conducted for teachers from each grade span/content area. 
These trainings each lasted approximately one half-day, and were conducted by a member of 
the evaluation team with expertise on the Common Core and the rating of student and teacher 
artifacts. Each training session included an overview of the LDC goals, template task, the 
structure of the modules, and the CRESST rating dimensions. These dimensions were each 
rated using a five-point scale and focused on 1) the effective writing task, 2) alignment to the 
CCSS and local and state literacy and content standards, 3) fidelity to LDC module instruction, 
4) quality instructional strategies, 5) the coherence and clarity of the module, and 6) the 


raters’ overall impression (see Appendix A for full descriptions of the dimensions). 


Once the training was complete, calibration was conducted by having teachers 
individually score and then discuss their ratings for one module in the content area in which 
they would be focusing. Scoring was then conducted on subsequent days with each module 
individually rated. All modules that were rated by at least two expert teachers were then 
discussed, with the goal, but not the requirement to reach consensus (see Carlson & McCaslin, 
2003). All discussions were facilitated or supervised by the same member of the evaluation 


team who conducted the initial training. 


Because of the relatively small number of total eligible modules and the speed at which 
panelists were able to calibrate, while we initially planned for one-third of the modules to be 
analyzed by more than one rater, the final rate of crossover was 49 or 92% of modules. It 
should also be noted that four of these modules were rated by all three members of the 
Secondary ELA panel as part of their calibration activities. 


Table 2.3 presents a brief overview of modules rated. More specifically, 62% of the 
modules were at the elementary level with the remaining modules spanning middle school 
and/or high school. Elementary modules were primarily focused on ELA or science, while 
secondary modules were more likely to focus on social studies. “One-week one-text” modules 
were a minority of both the elementary and secondary sets. Both elementary and secondary 
teachers were far more likely to adapt a module than create one that was original, and 
modules with one author were much more common than modules with multiple authors. While 
over two-thirds of the modules did provide background information for students, the inclusion 


of extension activities and teacher reflections were less common. 


17 


Table 2.3 
Background Variables for all Modules Rated (N = 53) 


Elementary Secondary Total 
Variables # % # % # % 

Module format 
One week, one text 9 27.3 9 45.0 18 34.0 
Multi week, multi text 24 72.7 11 55.0 35 66.0 
Module origin 
Adapted 25 75.8 15 75.0 40 75.5 
Original 8 24.2 5 25.0 13 24.5 
Author count 
Co-authored 8 24.2 4 20.0 12 22.6 
Sole 25 75.8 16 80.0 41 77.4 
Subject 

ELA 13 39.4 5 25.0 18 34.0 
Science 14 42.4 5 25.0 19 35.9 
Social Studies 5 15.2 9 45.0 14 26.4 
Other 1 3.0 1 5.0 2 3.8 
Module components 
Student background 23 69.7 15 75.0 38 71.7 
Extension activities 18 54.6 3 15.0 21 39.6 
Teacher reflection 7 21.2 12 60.0 19 35.9 
Total 33 62.3 20 37.7 53 100.0 


2.4 Survey Recruitment and Administration 


We began recruitment for the study by attending and soliciting consent forms at LDC 
Launch Days at the beginning of the 2016-17 school year. The consent forms included 
language stating that the study was voluntary, all data would be protected, and that by 
signing the form, participants gave their consent to be emailed an electronic survey in spring 
2017 and their permission for CRESST to request district data that link the teacher to 
students. All participants - including teachers, project liaisons, and administrators - were 
promised a $50 gift card for completing the survey in the spring. In the pilot year, teachers 
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and project liaisons were offered a $20 gift card, and administrators were not offered any 
incentive. The amount was increased and eligibility widened so as to increase buy-in and 
participation in the study. The higher incentive award, as well as refinements to recruitment 
and survey completion protocols described below, raised the overall survey response rate to 


78%, up from 57% the prior year. 


Most LDC participants were unable to attend the Launch Days at the beginning of the 
year so CRESST used other methods to recruit and collect consent forms. With LDC staff 
assistance, a CRESST staff member participated in a common planning session for each PLC 
either in person or via videoconference. For sessions joined via video conference, consent 
forms were mailed to project liaisons in advance and project liaisons returned completed 
consent forms in a prepaid envelope or by scanning and emailing them back to CRESST. 
CRESST staff joined common planning sessions from October 2016 through June 2017. 
CRESST also followed up with individual participants to collect consent forms, but this method 


was less successful than recruitment during launch days and PLC sessions. 


Surveys were administered in spring 2017. CRESST coordinated with LDC staff and 
coaches to administer the online surveys during common planning sessions. Multiple email 
reminders were sent to participants who were not present at the common planning session 
or who otherwise did not complete the survey. The teacher survey was closed at the end of 
the school year in June 2017. Administrator surveys were left open until early August when 
LDC staff met with administrators to plan for the 2017-18 school year. 


2.5 Analytical Approaches 


Multiple analytic procedures, including both quantitative and qualitative analytic 
methodologies, were applied to the data to answer evaluation questions about how LDC was 
implemented, conditions affecting implementation, and program impacts. The following 


sections describe the methodological approaches used to analyze the collected data. 


Surveys. Descriptive statistics were generated for multiple-choice survey questions. 
Open-ended responses were reviewed for key themes and summarized. We collected 
responses from three groups of survey respondents: teachers, project liaisons, and 
administrators. Individuals who played more than one role are represented in the results for 
both roles. Therefore, the samples for the different analyses are not mutually exclusive. 
Generally, we report the number and percentage of respondents who selected different 
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multiple-choice options. In a few cases where responses are numerical rather than 


categorical, we present means rather than proportions. 


Means and frequencies for all multiple-choice responses are presented in full in Appendix 
B for teachers, Appendix C for project liaisons, and Appendix D for administrators. Responses 
are organized by domain and specific questions are identified by question number. For 
example, the domain of LDC participation is summarized in one section, but relies on 
information from four questions. We preface teacher questions with “T,” project liaison with 
“PL,” and administrator questions with “A”; please refer to the corresponding appendices for 


complete results on all items. 


LDC CoreTools. The first stage of our analysis examined the proportion of all 
participants who created CoreTools user accounts, and engaged in three key behaviors: 
viewing, editing, and commenting on modules. We then analyzed each of the three key 
measures of participants’ interaction with the LDC online tools, and reported means and/or 
frequencies as measures of participants’ engagement with the online LDC system. We 
calculated and presented total, average, maximum, minimum, and standard deviation for 
each of the measures in the dataset. In addition to reporting the overall results, we also 
provided results by the content subject the teacher taught (ELA, social studies/history, etc.), 
by participant role (project liaison or teacher), and school level (elementary, K-8, middle, and 
high) whenever feasible. Finally, we explored the difference in CoreTools engagement 
between two groups of teachers: those that completed and taught modules and those that 


did not complete the design and implementation process. 


Modules. We used descriptive statistics (means, standard deviations) to analyze 
overall and subgroup performance for each content area/grade span on each of the six 
dimensions. Additionally, for the elementary modules, which had a moderate sample size, we 
used generalizability theory (G theory) to examine potential sources of error during the rating 
process to help determine the validity of the scores as well as the construct validity of the 
rubrics (see Shavelson & Webb, 1991). Principal components factor analysis was also 
conducted to determine whether the six dimensions examined together represent one 
dominant or underlying trait for the elementary sample. Finally, teacher comments during the 
debriefings for both elementary and secondary module ratings were examined to determine 


other potential issues with the rubrics and/or rating process. 


Generalizability theory is a statistical framework for examining multiple sources of 
potential error during the scoring process. For each sample, we first modeled score variability 


across all six dimensions using a two-faceted design, whereby we estimated variance 
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components for module by rater by dimension (t*r*d). The goal here was to separate true 
variation in the quality of modules from other potential sources of measurement error. The 
main effects reflect true variation across modules (o2t) and error variance across raters (o2r) 
and dimensions (02d), while the error term (o2trd,e) reflects unexplained residual error in 
the model. To disentangle the sources of potential error further, we also used a single-faceted 
design to examine potential error within the scoring of each dimension. As with the first set 
of models, the main effect reflects true variation across teachers (o2t) and error variance 


across raters (o2r). 


Student Outcomes. We employed a quasi-experimental design to examine the effect 
of LDC on the Smarter Balanced Assessment Consortium (Smarter Balanced) ELA assessment 
scores of students in the participating LDC elementary and middle schools in 2016-17. Before 
conducting the analysis, we used a two-step matching process to identify a reduced pool of 
comparison students and teachers within schools with similar characteristics to the schools in 
the intervention sample. 


To accomplish this, we first identified the five most similar control schools for each 
intervention school based on a Euclidian distance measure, by using the nearest neighbor 
analysis option in SPSS 24.0 (see Fix and Hodges, 1951; Wang, Neskovic, & Cooper). The 
variables used in this process were the percentage of students eligible for free or reduced 
price lunch, the percentage of African American students, mean prior student achievement in 
ELA, mean prior student achievement in math, the average attendance rate of teachers, the 
percentage of teachers with three or fewer years of teaching experience, and the school grade 
span where feasible. Once the pool of comparison schools was identified, their students and 
teachers were also identified and student-level matching was conducted so that the resulting 
sample would resemble the type of sample one would expect to obtain through random 
assignment. 


The student-level matching technique we employed was Coarsened Exact Matching 
(CEM) (Iacus, King & Porro, 2011). CEM is a flexible matching approach with many favorable 
properties, and allows the researcher to specify the precise conditions under which students 
are matched. For categorical variables, such as race/ethnicity or free or reduced price lunch 
status, this often entail exact matching, while for continuous measures, such as prior 
individual student achievement and aggregate class level achievement, cut-points for 
matching can be specified. With this approach we were able to set precise cut-points on the 
most important prior indicators, such as prior academic achievement, to ensure that where 
possible every treatment student was matched with a suitable comparison. Student matching 


variables we used in CEM included Hispanic, Black, poverty status, female, English language 
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proficiency (English Language Learner), special education status, gifted status, mean prior 
achievement in math and ELA, and grade level. 


During matching we also included a few variables capturing information on the teachers 
and peers to which students were exposed. These variables included mean prior ELA 
achievement of the student’s peers in her core content classes, the mean attendance rate of 
the student’s core content teachers, and the percentage of the student’s core content teachers 


who had three or fewer years of teaching experience. 


The typical structure of course taking at the middle school level involves students 
potentially being exposed to multiple teachers, with each responsible for a different core 
content class. Specifically, middle school students in the study had exposure opportunities 
across three content areas taught by intervention teachers (ELA, social studies/history, and 
science). As a result, students were not necessarily nested under individual teachers, but 
instead were likely to have connections to multiple teachers in the time period prior to each 
testing outcome (students at the elementary school level were also sometimes exposed to 
multiple teachers but to a lesser extent). Therefore, LDC effects were estimated using an 
extension of the standard multi-level modeling framework known as Multiple Membership 
Multiple Classification (MMMC) models (Browne et al., 2001). 


These models can account for complex classification structures, such as the LDC context, 
in which students are nested within schools but are also members of multiple classes led by 
different teachers who may or may not be implementing LDC. MMMC has the flexibility to 
account for this type of complex nesting structure in which students are hierarchically nested 
under schools but may have one-to-many relationships with teachers. As can be seen in Figure 
2.1, there are three classification levels in the models we employ: students, teachers, and 
schools. In the MMMC modeling approach, each observation at the lowest level represents 
one student. The double arrows linking students to teachers in Figure 2.1 signifies the 
possibility of one student being exposed to multiple teachers. The single arrow from teachers 
to schools signifies that teachers were nested with schools. 
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Students 


Figure 2.1: Multiple Membership Multiple Classification Structure 


In the MMMC modeling approach, each observation at the lowest level represents one 
student. The weight each teacher receives with respect to each student is dependent on the 
student’s exposure to his or her teachers in each of the three core content areas. The total 
student exposure adds to a unity (i.e., a possible total exposure of 1) across their courses in 
the three content areas in a given school year. While this general weighting approach applies 
to both elementary and middle schools, the course structure of the datasets required us to 
use different weighting procedures in elementary and middle school. 


In the elementary data set, students were linked to teachers through the course marks 
they received. Students received up to 14 different course marks across each of the three 
grading periods. We then collapsed the course links into the three core content areas of 
interest of our study (see Appendix F for details on the courses included). For example, course 
marks in reading, listening, writing, and speaking were all collapsed into the ELA content area. 
In the event that a student was exposed to more than one teacher, each content area was 
given equal weight in distributing teacher/student exposure. If a student completed all three 
grading periods of ELA and all three grading periods of social studies/history under one 
teacher, then that teacher was coded as 0.67 for having contributed to two-thirds of the 
students’ core curriculum exposure. If the same student received course marks in science 
from a different teacher than the one who was linked to their course marks in ELA and social 
studies/history, then that science teacher would have been coded as 0.33 and all other 
teachers in the sample would have been coded as zero. This would then result in the student’s 


exposure adding to a unity (1). 
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In middle school, students had two semesters of exposure to teachers in the three core 
content areas. Each full exposure within a semester counted as one unit in our model and 
exposure to teachers was measured in these units of instruction. For example, most 
commonly a student had two units of core instruction exposure in each the three content 
areas (two units of ELA instruction, two units of social studies instruction, and two units of 
science instruction) for a total of six units. In that scenario, if a student had exposure to three 
different teachers, with each contributing two units, then each teacher would contribute one- 
third (0.33) of the overall core curriculum exposure and all other teachers in the sample would 
be coded as zero. In addition to core ELA courses, supplemental ELA courses were also 
included in the LDC analysis, which made it possible then for a student to accumulate more 
than six units in the three content areas. The weighting in middle school was always 
distributed as a proportion of the total units in the three content areas. Therefore, if a student 
accumulated two core ELA units, two supplemental ELA units, two social studies units, and 
two science units, the base number of instruction units would be eight. If, using that same 
scenario, the same teacher taught both the core and supplemental ELA units then that teacher 
would contribute one-half (0.50) of the overall core curriculum exposure with the social 
studies and science teachers contributing one quarter (0.25) each, again resulting in the 
student’s exposure adding to a unity (1). 


For this study, we modeled the treatment intervention variable as a fixed effect at the 
student-level in two different ways. The first dosage dependent approach takes into account 
the students’ level of exposure to the intervention teachers. In this approach, the treatment 
was structured as a continuous response variable, coded as zero for comparison students and 
coded as a positive value for treated students, albeit, never exceeding one. The positive value 
assigned to treated students in the dosage dependent approach was simply the sum of the 
intervention teacher weights linked to the treated student. The second approach was dosage 
independent and classified any student exposed to an intervention teacher via at least one 
course as a treated individual. In this approach the treatment variable was dichotomous 


(coded as one for treated students and zero for comparison students). 


As with other multi-level models, MMMC models account for the non-independence of 
observations within cluster by adjusting the inferences on parameter estimates for the 
correlations between responses in a cluster. This modeling approach, however, becomes 
computationally cumbersome using traditional frequentist estimation methods. As 
recommended by Browne et al. (2001) we instead employed Bayesian methods using Monte 
Carlo Markov chain (MCMC) techniques to best address this issue. Multi-level models 


incorporate demographic and achievement variables used in the matching design as 


24 


covariates, making the findings “double robust” (characteristics controlled for in both 
matching and outcomes analysis stages). Student demographic and prior achievement 
variables that were used in the matching process were also included as covariates in the 
MMMC model. In the elementary analysis, mean peer prior achievement, teacher attendance, 
and teacher experience were not significant so they were not included in the final elementary 
models. In addition, for the middle school results, the number of supplemental ELA course 
units was insubstantial so those course exposures were not included in the final middle school 
models. The full specifications for both models can be found in Appendix F. 


25 


3.0 Survey Analysis 


Five groups of participants were surveyed: (1) Teacher, (2) Teacher/Project Liaison, (3) 
Project Liaison, (4) Administrator, and (5) Administrator/Project Liaison. Twenty Cohort 1 
schools began implementing the LDC program in the 2016-17 school year, with 154 classroom 
teachers participating and 34 administrators overseeing the work (see Table 3.1). The 20 
schools included 11 elementary schools, four middle schools, one high school, two K-8 
schools, one 6-12 school, and one K-12 school. Participants taught across all grade levels 
from K to 12. Most secondary teachers taught ELA, social studies/history, or science, with a 
handful teaching other subjects such as math, foreign languages, special education, or the 


arts. 

Table 3.1 

Survey Consent and Response Rates: 2016 

N of 
N of N of Survey Consent Response 
Participant Type Participants Consents Responses Rate Rate 

Teacher 139 127 110 91% 79% 
Teacher/Project Liaison 15 15 12 100% 80% 
Total Teachers 154 142 122 92% 79% 
Project Liaison 5 5 4 100% 80% 
Administrator 27 21 19 78% 70% 
Administrator/Project Liaison 2 2 2 100% 100% 
Total Administrators* 34 28 25 82% 74% 
Total Participants 188 170 147 90% 78% 


*We categorize the coaches and coordinators who completed the Project Liaison survey as 
administrators. 


Table 3.1 presents the study consent and survey response rate information. As shown, 
survey consent and response rates were different for teachers who implemented LDC in their 
classrooms and administrators. Compared to administrators, teachers had a considerably 
higher consent rate (92% compared to 82%) and survey response rate (79% compared to 
74%). Teachers who were project liaisons were more likely to consent to participate in the 
study but no more likely to complete the survey. Overall, consent and response rates were 
quite high and were markedly improved from the pilot year of the study. 
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We discuss survey results according to the domains listed in Table 2.1. We also include 
specific questions covered in each domain for easier reference. For example, “LDC 
Participation (T1-4)” indicates that teacher questions 1-4 are used to provide information on 
LDC participation. Multiple choice survey questions and descriptive results (frequencies and 
means) are presented in full in Appendix B for teachers, Appendix C for project liaisons, and 
Appendix D for administrators. As noted earlier, these three samples are not completely 
mutually exclusive (i.e., some teachers and administrators also acted as project liaisons). 
Results are organized by domains and question number. For example, the domain of “LDC 
participation” is summarized in one section, and relies on information from four different 
questions. We preface teacher questions with “T,” project liaison with “PL,” and administrator 
questions with “A.” 


Survey results are presented in four sections. The first section summarizes teachers’ 
responses. The second section summarizes project liaisons’ responses, and the third section 
summarizes administrators’ responses. Whenever we felt a comment from an open-ended 
response might clarify, illustrate, or corroborate a finding, we included that comment in the 
appropriate section. The fourth section summarizes open-ended responses from. all 
participants, who answered the same four questions about program efficacy and 
improvement. We end with a summary of results. 


3.1 Teacher Survey Results 


A total of 122 teachers at 19 schools completed the survey (Table 3.2). One participating 
school did not return any teacher surveys. Of the 122 teachers who responded to this survey, 
52% taught in ten elementary schools, 24% in four middle schools, 6% in two K-8 schools, 
and the remaining 28% of teachers taught in three schools each with a different grade range 
(one high school, one middle/high school, and one K-12 school). Six teachers (5%) indicated 
they had not participated in a professional learning community in the 2016-17 school year. 
Therefore, they were skipped past most of the survey questions. In the following sections, we 
report on each survey domain. In addition to producing descriptive statistics on the whole 
teacher sample, we also produced results for elementary and secondary level teachers. We 


highlight important differences between the two subgroups, when they are apparent. 
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Table 3.2 
Participating Schools and Teachers in 2016-17 


School Type N of Schools N of Teachers % of Teachers 
Elementary Schools 10 63 52% 
K-8 Schools 2 7 6% 
Middle Schools 4 29 24% 
Middle/High Schools 1 4 3% 
High Schools 1 16 13% 
K-12 Schools 1 3 2% 
Total 19 122 100% 


LDC Participation (T1-4) 


Almost all teachers (92%) reported that this year was their first experience with LDC. 
The remaining 10 teachers who had prior experience with LDC reported that they had taught 
0 to 6 modules (mean=1.8) and O to 6 mini-tasks (mean=2.2) outside of modules. Three of 
these 10 teachers taught zero modules, and four of these teachers taught no mini-tasks. 


Among the 122 teachers who completed the survey, 69 of them taught in elementary 
grades, and 53 taught secondary grades. The secondary teachers reported teaching 1 to 14 
classes (mean=5.3); in these classes, they used LDC materials in 0 to 7 classes (mean=2.9) 
(Two teachers reported not using LDC materials in any of their classes). Over a third (37%) 
reported using LDC in ELA; 31% in history/social studies; 21% in science; 4% in 
mathematics; and 8% in other areas (health, Spanish, etc.) (see Figure 3.1). 


Content Area Implementations 


40% 37% 
35% 31% 
30% 
25% 21% 
20% 
15% 
10% 8% 

0% || 

ELA (n=19) History/Social Science (n=11) Math (n=2) Other (Spanish, 


Studies (n=16) health) (n=4) 


Figure 3.1. Secondary Teachers’ Content Area Implementations 
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Professional Learning Community and Teacher Collaboration (T5-9, T30, T37) 


Almost all teachers (95%) participated in a PLC that was at least partly focused on 
implementing LDC. The six teachers who had not participated in a PLC reported they had used 


LDC tools in their planning or instruction. 


Almost a third (30%) of teachers reported meeting in their LDC PLC at least once a week 
or more. Almost half (46%) met every other week, and 24% met once a month. Of the 19 
schools, teacher survey responses suggested that five met approximately once a week or 
more; 10 met every other week; and four met once a month. The most common reason cited 
for not meeting weekly was that PLC members had other priorities (59%). Other reasons cited 
were technology issues and the fact that teachers were not paid for the time. Interestingly, 
secondary teachers were considerably more likely to meet just once per month (42% 
compared to 11% of elementary teachers) and much less likely to meet at least once per 
week (10% compared to 46% of elementary teachers). 


According to teachers, LDC PLC meetings most often lasted 45 minutes to an hour 
(72%). About a quarter of teachers (23%) reported that meetings lasted longer than an hour, 
and 5% less than 45 minutes. Most teachers (70%) agreed or strongly agreed that their PLCs 
were given sufficient time to meet. However, in the open-ended responses, 64 teachers (55% 
of 116 teacher commenters) indicated that time was a barrier to effective implementation. 
These comments included problems with time other than insufficiency, such as regularity of 
meetings and the need for teachers to have time outside of meetings to discuss and 


implement. 


A third of teachers (33%) said they had informal discussions about LDC with their 
colleagues once a week or more; 32% every other week; 17% once a month; and 18% less 
than once a month. These groupings generally corresponded with the school breakdown for 
frequency of formal meetings. In other words, the more frequently a school’s PLC met, the 
more likely it was for that school’s teachers to also engage in informal discussions outside 
PLC sessions. As with formal meetings, secondary teachers reported interacting in informal 
settings less frequently than elementary teachers. Over three quarters (77%) of teachers 
agreed or strongly agreed that they were more likely to collaborate with other teachers on 
designing instruction after participating in LDC; 69% agreed or strongly agreed that LDC 
participation helped them develop working relationships with teachers in different grades or 


subjects. 
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LDC Training and Support (T10-T13) 


Teachers evaluated the three main types of LDC training and support: professional 
learning community (PLC), online course materials, and virtual coach support during and 
outside of PLCs. 


Teachers found PLCs to be moderately effective or very effective in the following ways: 
creating an environment in which teachers were comfortable working with each other (83%); 
demonstrating the usefulness of LDC (74%); fostering an environment where teachers shared 
instructional plans with colleagues (74%); allowing space for shared problem solving (70%); 
and helping teachers learn to develop modules (68%) (see Figure 3.2). Elementary teachers 
were a bit more likely to report that their PLC was effective. For example, 79% of elementary 
level teachers said that their PLC was moderately effective or very effective in demonstrating 
the usefulness of LDC, compared to 68% of secondary teachers. 


90% 83% 

80% 74% 74% 70% Bex 
70% 

60% 

50% 

9 ty) 
se 26% 26% ane = 
20% 17% 

PRR ili 
0% 
Demonstrating the Creating an Fostering an Allowing space for Helping teachers 
usefulness of LDC environment in environment shared problem _ learn to develop 
for teachers (N=  whichteachers whereteachers solving (N=116) modules (N = 116) 
116) were comfortable shared their 
working together instructional plans 
(N = 116) (N = 116) 


Moderately effective/Very effective ONot Effective/A little effective 
Figure 3.2. The Effectiveness of Teacher’s PLC (T10) 


Different aspects of the online course materials were rated as good or excellent by a 
majority of teachers in almost all cases including: relevance of information (68%); usefulness 
of resource documents such as the LDC Pitfall Checklist (56%); usefulness of videos (52%); 
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helpfulness in creating modules (51%); opportunity to extend learning (59%); and ease of 
use (47%) (see Table 3.3). 


Table 3.3 
Rating Online Course Materials (n = 116) 


Poor Fair Good Excellent 


How would you rate each of the following aspects of the online course material 
(in the Learn tab in LDC CoreTools) that your coach used or directed you to use? 


Clarity of information presented 9% 26% 48% 17% 
Relevance of information presented 8% 24% 47% 21% 
Ease of use 16% 37% 38% 10% 


Usefulness of resource documents (e.g., LDC Pitfall 
Checklist, CCSS Mental Markers, etc.) 


Usefulness of videos 17% 31% 40% 12% 


16% 28% 44% 12% 


Degree to which course material helped me to create 


(0) oO oO Oo 
and/or adapt LDC modules 11% = 38% = 39% 12% 


Opportunity to extend learning when needed or 


; 14% 28% 43% 16% 
desired 


Most teachers (91%) said they were able to get the feedback and support they needed 
from their LDC coach and that coaches provided written feedback in a timely manner (87%). 
In addition to direct support during PLCs, the following types of asynchronous coach support 
were found to be moderately or very helpful by about half or more of teachers: written 
feedback in CoreTools (62%; 21% did not use); individual Zoom conference (60%; 22% did 
not use); email or phone communication (51%; 32% did not use), and facilitated discussion 


on Teaching Channel Teams (43%; 35% did not use). 


LDC Implementation (T14 -28, T30) 


This domain covers questions on module creation (T14-17), module peer review (T27- 
28), and classroom implementation (T18-26, T30). 


Module Creation (T14-17) and Module Peer Review (T27-28). Teachers adapted 
or created two types of LDC instructional products: mini-tasks are short, generally taking one 
period, and focus on a specific skill; modules are longer, more complex units comprised of 


multiple mini-tasks, ending in a culminating “teaching task.” 


Using existing LDC templates and exemplars, teachers individually or collaboratively 


adapted between 0 and 12 modules during the year (mean=1.6). Among the 104 teachers 
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who answered this question, 17 teachers (16%) adapted no modules, 80 teachers (77%) 
individually or collaboratively adapted between one and three modules during the year, six 
teachers (6%) adapted four to six modules, and one teacher (1%) reported adapting 12 
modules. The mean number of modules teachers adapted was 1.6 modules. Creating modules 
from scratch, teachers individually or collaboratively constructed 0 to 4 modules (mean=1.1). 
Twenty-eight teachers (27%) constructed no new modules, 67 teachers (55%) individually 
or collaboratively constructed one or two modules, and nine teachers reported (8%) either 
three or four modules. Among these 145 teachers, the mean number of modules created was 
1.1 modules. Elementary teachers chose to adapt modules more often than secondary 
teachers, while secondary teachers were much more likely to create new modules from 
scratch (27% of secondary teachers reported adapting zero modules and 40% of elementary 
teachers reported creating zero new modules). 


Teachers reported how they constructed modules. Some teachers created more than 
one module during the year using different collaboration structures. Half of teachers (50%) 
worked in teams of two or more to create modules. Over a third (36%) created modules 
individually. Less than a quarter (23%) wrote modules with the entire PLC. Ten percent of 
teachers reported "other," with a few specifying that they did not write modules at all or that 
they wrote mini-tasks instead. 


Reflecting on the instructional strengths of their modules, most teachers felt confident 
in their ability to execute all eight of the instructional features of LDC modules that were 
tracked: teachers reported being able to select focus standards for the writing assignment 
(88% felt they were able to accomplish this to a moderate or great extent); create a 
standards-driven writing assignment (83%); identify the skills needed to complete the 
assignment (78%); create an assignment that provided multiple opportunities for student 
engagement (77%); select high quality texts (73%); make connections to previous or future 
learning (68%); create daily skills lessons (65%); and plan multiple methods to assess 
progress (61%). 


Of these eight features of module creation, the first three elements in Table 3.4 below 
had the highest percentage of teachers who felt they had accomplished it to a great extent, 
indicating a higher level of confidence than for the other aspects of assignment planning. 
Those skills were selecting focus standards (47% felt they had accomplished this to a great 
extent); creating a standards-driven assignment (38%); and providing multiple opportunities 
for students to engage (31%). For the other five elements, the percentage of teachers who 
felt they had achieved it to a great extent ranged from 15% to 27%. 
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Table 3.4 
Rating on Teacher Skills Associated with Creating LDC Modules and/or Mini-tasks (n = 116) 


Not at A little a A great 
all : moderate extent 
bit extent 


Please indicate to what extent you were able to do each of the following when 
creating LDC modules and/or mini-tasks. 


Select a set of focus standards for a writing assignment 3% 9% 41% 47% 
Create a standards-driven writing assignment task 4% 13% 45% 38% 
Select high quality, complex texts and other materials 


10) 10) 10) 10) 
that allowed students to engage in deeper learning 2 ees aoe ae 
Create a writing assignment that provided multiple — 9% 15% 46% 31% 
opportunities for students to engage with the material 
Identify the skills students need to develop in order to 6% 16% 52% 27% 
complete a writing assignment 
Create daily lessons to teach the skills a student needs 9% 27% 44% 21% 
to complete a writing assignment 
Plan for a variety of methods to assess student 6 6 " é 
progress (e.g., mini-task scoring guides) a sue a1 15% 
Make connections to previous or future learning that 9% 23% 46% 22% 


make a writing assignment relevant for students 


Classroom Implementation (T18-26, T30). After creating their modules, teachers 
implemented modules in their classrooms and reflected on their progress. Teachers 
implemented O to 6 modules (mean=1.7) and implemented from O to 20 mini-tasks 
(mean=2.6) that were not part of modules (see 
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Table 3.5 for more information). A very substantial proportion of teachers (54 out of 
116) therefore did not implement the recommended 2 modules over the course of the school 
year. Teachers submitted 0 to 4 modules (mean=0.7) for LDC National Peer Review. Of the 
54 teachers who reported submitting modules for peer review, four (7%) found the process 
very helpful, 19 (35%) moderately helpful, 19 (35%) a little helpful, and 12 (22%) not helpful. 
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Table 3.5 
Number of Modules and Mini-Tasks Implemented (n = 116) 


Numbers of Modules or Mini-tasks N of Teachers N of Teachers 
implemented in 2016-17 Implementing Implementing 
Modules Mini-Tasks 

0 10 23 

1 44 24 

2 44 32 

3 14 13 

4 2 9 

5 3 

6 2 2 

7 1 

8 2 

10 4 

14 2 

20 1 


Teachers reflected on their ability to provide instruction through the modules. Across all 
six dimensions, the distribution of teachers who felt they were able to implement effective 
instructional strategies through the modules was high: 78% felt they had engaged students 
in productive struggle to a moderate or great extent; 77% differentiated instruction; 72% 
located performance of standards in student work; 69% used evidence of learning to modify 
instruction; 68% shared expectations of learning with students; and 65% systematically 
collected information about student learning. Also across all six dimensions, the percentage 
of teachers who felt they had accomplished these strategies to a great extent ranged from 
22-27%, indicating that about a quarter of teachers felt very confident about their ability to 


implement high quality instructional strategies using LDC modules. 


The One-Text One-Week module was introduced as a first step toward module creation. 
As its name indicates, the module highlighted one text that was examined closely over one 
week. Fourty percent of teachers taught a One-Text One-Week module. Of those teachers, 
61% created the module from an LDC template, while the remaining 39% copied and adapted 
another teacher’s module from the LDC online library. 


Following the One-Text One-Week modules, teachers typically worked on a longer-term 
instructional module. Just over half of teachers (53%) copied and adapted another teacher’s 
module from the LDC online library, while 47% created a module from an LDC template. 
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Elementary teachers were more likely to have adapted an existing module (about two thirds 
did so), while secondary teacher were more likely to have created new modules from a 
template (again about two thirds of the subgroup). Over two thirds (68%) of teachers had 
already taught this module during the school year at the time of the survey administration, 
while 15% planned to teach it before the current school year ended, 14% planned to teach it 
the following school year, and 3% had no plans to teach the module. 


Leadership Support (T32-36) 


This domain covers questions on project liaison support and administrator support (T32- 
33) and teacher leadership role in LDC (T36). 


Project Liaison Support (T32). Project liaisons were school staff, either participating 
LDC teachers or school administrators, who provided logistical support to the PLCs. Responses 
indicated that the vast majority of teachers were very satisfied with the level of project liaison 
support. The project liaisons were approachable (90% agreed or strongly agreed); gave useful 
feedback (77%); effectively supported the PLC meetings (84%); had a strong grasp of LDC 
(85%); helped teachers align LDC to broader school goals (79%); and effectively invited 
teachers to join LDC (78%). Again, there was a difference between elementary and secondary 
teachers, with secondary teachers somewhat less likely to express confidence in their project 


liaison. 


School Administrator Support (T33-35). School administrators were principals, 
assistant principals, or other instructional leaders who observed teachers in action and 
provided other leadership, such as protecting time for PLCs to meet. Teachers provided 
feedback about the support they received from their school administrator. Almost all teachers 
(94%) agreed or strongly agreed that their administrator encouraged LDC participation. 
According to teachers, most administrators were able to ensure PLCs had time to meet (79%); 
communicate how LDC supported school initiatives and goals (75%); and had a firm 
understanding of LDC (72%). Most administrators reportedly also made formative assessment 
a priority (69%). Less than half reported that their administrator had provided feedback about 
LDC planning and instruction (47%). A minority of teachers reported that administrators 


expressed concern that LDC was taking time away from other instructional priorities (41%). 


The involvement of school administrators, as reported by teachers, was varied. Over a 
third (35%) attended more than three quarters of PLC meetings, but a similar number (28%) 
attended less than one quarter. Ten percent attended about one quarter, 17% attended about 
one half, and 10% attended about three quarters. The majority of teachers (58%) reported 
never being observed by an administrator while teaching an LDC mini-task. A quarter (25%) 
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reported being observed once; 10% were observed twice; and the remaining 8% were 
observed three or more times. 


Teacher Leadership Role (T36). The role of teachers in LDC extends beyond the 
classroom. As teacher leaders, LDC teachers are pivotal in driving LDC implementation in their 
schools. Most teachers agreed or strongly agreed that LDC helped them create writing 
assignments within their current curriculum (78%). About half of teachers felt they were able 
to affect the direction of LDC at their site by having the opportunity to work with the project 
liaison and administrator (54%); setting instructional goals for LDC work (53%); and being 
involved in discussions about expanding LDC implementation in future years (48%). Over half 
of teachers (59%) expressed interest in learning more about facilitating LDC implementation 
at their schools. Here as well differences were observed between school levels. While a 
majority of elementary teachers felt empowered by the LDC experience, less than half of 
secondary teachers indicated that they had played or would like to play a leadership role in 
LDC. 


LDC Impact (T29-31) 


This domain covers questions on LDC impact on teacher practice and learning (T29-30) 
and student learning (T31). 


Impact on Teacher Practice and Learning (T29-30). We asked teachers about how 
LDC had changed their ability to practice key teacher skills. Specifically, we asked them to 
focus on the change from the beginning to the end of the current school year’s work with 
LDC. We also asked them to provide more information about the impact of LDC on their 
instructional practice. 


Over all eight items concerning change in ability during the year, about two thirds to 
three quarters of the teachers rated themselves as having improved moderately or a great 
deal. Specifically, the areas teachers were most likely to perceive improvement concerned 
instructional planning: identifying skills needed to complete an assignment (77%); selecting 
focus standards (76%); and creating high quality writing tasks (71%). The other areas of 
impact on teacher practice and learning for which teachers were less likely to perceive impact 
involved collecting and using student data to inform instruction: assessing student progress 
(66%); using evidence of student learning (66%); tracking and analyzing student progress 
(61%); and identifying student misconceptions (60%). These self-ratings correspond with the 


high emphasis placed during PLCs on designing instructional units. 
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| often incorporated LDC mini-tasks into my 
non-LDC instruction. (n=116) 


Participating in LDC helped me develop . 
working relationships with teachers in ———________, 69% 
(n=116) ° 


different grades and/or subjects. 


lam more likely to collaborate with other 5 
Rearheieerseslenipe etrus\lon aftr irs 775 
(o} 


participating in our LDC Professional Learning 


LDC helped me incorporate writing 5 
sssignments Into my existing curricu!4m, ul aldiee mm 2: 
0 


(n=116) 


Implementing LDC helped me incorporate my 5 

state’s College- and Career-Ready Standards 32% 
: : . 68% 
into my instruction. (n=116) 


Using LDC modules became an important part 41% 
of my instructional practice. (n=116) 59% 
Participating in LDC raised my expectations for 23% 
students’ writing. (n=116) 77% 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 


52% 
48% 


| 


O Disagree/Strongly disagree B Agree/Strongly agree 
Figure 3.3. Impact on Teacher Practice and Learning (T30) 


The areas of widest reported LDC impact on teacher practice (see Figure 3.3) included 
helping teachers incorporate writing assignments (78% agreed or strongly agreed); likelihood 
of teacher collaboration (77%); and raising expectations for student writing (77%). LDC also 
had slightly narrower but still apparent impact in the following areas: development of 
relationships with teachers outside grade or subject (69%); incorporating state standards 
(68%); establishment of LDC as a part of existing instructional practice (59%); and 
incorporating LDC mini-tasks into non-LDC instruction (48%). 


Impact on Student Learning (T31). Teachers were asked to evaluate the effect of 
LDC on student learning (see Table 3.6). On average, about three quarters of teachers agreed 
or strongly agreed that LDC impacted student learning in the following ways: supported 
students to complete writing assignments (78%); developed skills needed for college and 
career readiness (78%); increased student capacity to analyze components of the writing task 
(77%); improved overall literacy performance (77%); increased content knowledge (74%); 
developed speaking and listening skills (74%); improved quality of writing skills (72%); and 
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developed reading skills (68%). Across all items, secondary teachers were a bit less likely to 
agree that LDC had impacted student learning than elementary teachers. While three quarters 
or more of elementary teachers tended to report positive impact, typically only two thirds of 
secondary teachers did. 


Table 3.6 
Impact on Student Learning (n = 104) 


Strongly 
Disagree 


Strongly 


Disagree Agree Agree 


Please indicate the degree to which you agree or disagree with the following 
statements. 


LDC helped students develop 


Oo Oo 
reading skills. 10% 22% 60% 9% 


LDC was effective in improving 


ie) 18) 18) 10) 
students’ content knowledge. ne mee ote nee 


LDC modules effectively supported 
students in completing writing 8% 14% 64% 14% 
assignments. 


LDC was effective in improving the 


oO Oo Oo 
quality of students’ writing. 9% ue Sue oe 


LDC supported students’ 
development of skills needed for 8% 14% 66% 12% 
college and career readiness. 


LDC increased students’ capacity to 
analyze and understand the 


10) 
components of a writing 9% 14% 65% 12% 
assignment task. 
LDC helped students develop ‘ g ‘ . 
speaking and listening skills. 10% 16% 60% 14% 
Overall, LDC helped improve 90% ier ae ier 


students’ literacy performance. 


Facilitators and Barriers (T37) 


Successful implementation of LDC depends on a number of factors. We asked teachers 
to weigh in on the effect of these factors on implementation (see Figure 3.4). Over three 
quarters (77%) agreed or strongly agreed that their school had adequate technology to 
support teachers’ use of LDC. Most teachers (70%) felt their PLC was given sufficient time to 
meet, although the majority of teachers cited time as a barrier in the open-ended responses. 
Most teachers were able to easily find and adapt LDC mini-tasks indicating that they were 
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able to use the CoreTools online platform successfully (65%). Over half of teachers (60%) 
felt adequately prepared to implement modules in the classroom, leaving 40% who felt they 
needed more support. About half of teachers (51%) found it challenging to find content-rich 
texts to use with LDC modules. 


90% 


77% 
0, 
80% 70% - 
10, 
70% 60% 9 
60% 
. 51% 49% 
~ 35% 
409 
‘ 30% : 
30% 23% 
20% 
10% 
0% 
My LDC PLC was | felt adequately ‘It was challenging My school had It was easy to find 
given sufficient prepared to to find content-rich adequate and adapt LDC mini- 
time to meet. (N= effectively reading materials technology to tasks for use in my 
115) implement LDC for the LDC modules support teachers’ classroom. (N= 
modules in my | developed. (N = use of LDC. (N= 115) 
classroom. (N= 115) 115) 


115) 


@ Agree/Strongly agree ODisagree/Strongly disagree 


Figure 3.4. Facilitators and Barriers (T37) 


3.2 Project Liaison Survey Results 


This section presents the survey results from school-level project liaisons. At each 
school, the project liaison coordinated PLC meetings with LDC coaches. Project liaisons were 
participating teachers, school administrators, or other school support staff. A total of 18 
project liaisons responded to the survey. 


LDC Participation (PL1-2) 


Of the 18 project liaisons who responded, 16 (89%) reported that they did not have 
prior experience with LDC. 


Professional Learning Community and Teacher Collaboration (PL3-6) 


Less than half (39%) of project liaisons reported that their PLCs met once a week or 
more frequently. The 11 project liaisons whose PLCs did not meet weekly reported that the 
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primary barriers were that PLC members had other competing priorities (39%) and that PLC 
time was not protected (33%). 


Almost all project liaisons (83%) said these meetings lasted 45 to 59 minutes. A small 
minority (11%) said meetings were less than 45 minutes, and one liaison (6%) said they were 
60-74 minutes. Nearly a third (28%) reported that they had informal discussions about LDC 


with teachers once a week or more. 
LDC Training and Support (PL7-12) 


Project liaisons evaluated the effectiveness of the various training and support methods 
provided during LDC implementation. These included PLC meetings, online course material, 


and coach support. 


Project liaisons overwhelmingly reported that the PLC meetings were effective. More 
than three quarters of project liaisons (78%) thought PLC meetings were moderately or very 
effective in helping teachers learn to develop modules, and 72% said their PLC was effective 
in each of the following ways: created an environment in which teachers were comfortable 
working with each other; allowed space for shared problem solving; fostered an environment 
where teachers shared instructional plans with colleagues; and demonstrated the usefulness 
of LDC to teachers. 


Online course materials used by the coaches during PLC meetings also received high 
ratings from the project liaisons. Most project liaisons (83%) rated the clarity of information 
as good or excellent; 83% for relevance of information; 78% for ease of use; 78% for 
usefulness of resource documents; 78% for opportunity to extend learning when needed or 
desired; 67% for helpfulness to teachers in creating modules; and 61% for usefulness of 
videos. 


LDC coaches were highly regarded by project liaisons. All (100%) reported that they 
were able to get the feedback and support they needed from LDC coaches, and 94% reported 
that the coaches provided written feedback on modules in a timely manner. Different types 
of asynchronous coach support were rated by the majority of project liaisons as moderately 
or very helpful. These supports were: individual video conferencing with coaches (83% 
helpful; 6% did not use); email or phone communication (78% helpful; 6% did not use); 
written feedback in CoreTools (67% helpful; 22% did not use); and coach-facilitated 
discussions on Teaching Channel (33% helpful; 50% did not use). All project liaisons (100%) 
agreed or strongly agreed that they were able to reach their coach with questions; 95% said 
their coach responded quickly; 95% said the coach was knowledgeable and provided high 
quality guidance; 89% thought their coach was easy to work with; 89% said coaches 


41 


connected them with additional resources when needed; and only 33% of project liaisons 


reported that it was challenging to structure PLC time with the coaches. 


Project liaisons had the opportunity to attend LDC professional development meetings, 
both in person and online (these were in addition to PLC meetings). The average number of 
meetings attended was 3.4, with a range of 0 to 20. The majority (71%) participated in one 


or two meetings; 6% attended no meetings. 


Finally, project liaisons provided feedback about non-coach LDC support. A large Majority of 
project liaisons (78%) believed LDC offered sufficient professional learning opportunities to 
help them lead the initiative at their school, and 89% agreed that LDC provided adequate 
technical support for issues with the CoreTools online platform. 


LDC Implementation (PL 13-16, PL20) 
This domain covers questions on module creation (PL13-16) and alignment (PL20). 


Module Creation (PL13-16). Project liaisons reported the ways in which modules 
were created in their PLCs and judged the extent to which they were able to embed targeted 


instructional practices while creating the modules. 


Project liaisons were asked how many LDC modules were created by their PLC 
members, either individually or collaboratively by adapting them from existing modules 
created in a prior year or found in the LDC Library in CoreTools. Answers ranged from 0 to 2, 
with a mean of 1.4. Project liaisons reported that their PLCs created from scratch, either 
individually or as a group, 0-11 modules, with a mean of 1.9. Modules were most commonly 
created via a collaborative process, either by teams of two or more teachers (39%) or by the 
PLC as a whole (22%). 


In general, project liaisons were extremely confident in their ability to carry out 
targeted instructional practices while creating their modules. All project liaisons (100%) felt 
they were able, to a moderate or great extent, select focus standards for the writing 
assignment; 94% provided multiple opportunities for students to engage with the 
assignment; 89% identified the skills students needed to complete the writing assignment; 
83% were able to select high quality texts and other materials; 83% created daily lessons to 
teach the necessary skills; 78% made connections to previous or future learning to make the 
assignment relevant for students; and 61% planned for a variety of ways to assess student 


progress. 


Alignment (PL20). Project liaisons were asked how LDC aligned with other 


instructional priorities and programs at their schools. Alignment was generally perceived to 
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be quite high as reported in Figure 3.5. Nearly all (83%) agreed or strongly agreed that LDC 
complemented other initiatives at the school; 83% viewed LDC as a strategy for implementing 
statewide college and career-ready standards; 78% said LDC helped teachers create writing 
assignments to use within their existing curricula; 72% thought LDC helped prepare students 
for state assessments; 56% reported their school connected LDC to specific school goals; and 
only 50% reported that time spent on LDC interfered with other initiatives. Notably, however, 
almost three quarters (72%) believed that it was difficult for teachers to focus on LDC because 
of other competing priorities at the school. 


It was difficult for teachers to focus on LDC 


because of other competing priorities at the 28% 
72% 
school. (n=18) 
LDC helped prepare students in my school for 28% 
current state assessments. (n=18) tT _, 72% 


The time spent implementing LDC interfered 
with other important initiatives at my school. 
(n=18) 


50% 
50% 


| 


LDC was a strategy for implementing my 


state’s College- and Career-Ready Standards. 17% 
83% 
(n=18) 
LDC complemented other initiatives taking 17% 
place in my school. (n=18) ——————— 83% 


LDC helped teachers create writing 
assignments to use within their current 
curricula. (n=18) 


22% 


78% 


Our school connected LDC implementation to 44% 
our specific schoolwide goals. (n=18) 


| 


56% 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 


O Disagree/Strongly disagree @ Agree/Strongly agree 


Figure 3.5. LDC Alignment (PL20) 


Leadership Support (PL17-19, PL22) 


This domain covers questions on school administrator support (PL17-18), project liaison 
leadership role (PL19), and district support (PL22). 


School Administrator Support (PL17-18). Project liaisons reported on school 
administrators’ involvement with LDC. Over a third (38%) reported their administrators 


attended less than a quarter of PLC meetings; 25% about a quarter; 6% about half; 13% 
about three quarters; and 19% more than three quarters. 


All project liaisons (100%) reported their administrators encouraged teachers to 
participate; 81% said their administrators had a firm understanding of LDC; 69% reported 
their administrators made formative assessment a priority at their school; 69% felt planning 
time for LDC was protected; 63% said their administrator communicated how LDC supported 
other school goals and initiatives; 44% of project liaisons reported that their administrators 
expressed concern that LDC was taking time away from other instructional priorities; and 
38% said they received feedback about their LDC leadership from their administrator. 


Project Liaison Leadership Role (PL19). Project liaisons reflected on their leadership 
role in LDC implementation at their school sites. Overall, results were very positive and 
indicated that the majority of project liaisons felt engaged by and invested in the LDC 
implementation at their school. Almost all (88%) agreed or strongly agreed that they met 
regularly with their LDC coach to manage the work plan; 75% felt they were able to build 
capacity as an instructional leader as a result of their LDC leadership role; 69% helped set 
instructional goals for LDC implementation; 69% were involved in planning how to meet 
teacher learning needs by differentiating LDC implementation; 56% were involved in 
discussions about how to expand LDC implementation in future years; 56% believed their role 
as LDC project liaison helped them advocate for additional resources on their campus; and 
44% met regularly with their school administrator. 


District Support (PL22). Almost all project liaisons (83%) agreed or strongly agreed 
that district leaders supported implementation of LDC: 72% agreed that district leaders were 
interested in implementing LDC at other schools; 72% reported that district leaders visited 
their school to discuss LDC implementation; 67% thought district leaders had a firm 
understanding of LDC; and 56% said that district professional development efforts aligned 
with LDC. 


Scale-up and Sustainability (PL21) 


Regarding the future of LDC implementation, most project liaisons (72%) agreed or 
strongly agreed that they expected most teachers to continue with LDC the following year; 
72% felt teachers and administrators were committed to sustaining the initiative; and 67% 
expected their LDC PLC to increase in size the following year (see Figure 3.6). Generally, 
however, they did not believe LDC had yet reached beyond the PLC, with only 33% reporting 
that new collaborations were launched as a result of LDC and 28% reporting that non-LDC 


teachers used LDC resources, such as the planning process or CoreTools. 
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Figure 3.6. Scale-up and Sustainability (PL21) 


3.3 Administrator Survey Results 


This section presents the survey results from school principals and assistant principals. 
LDC Participation (A1) and Professional Learning Community (A2) 


Of the 21 school administrators who responded to this survey, 14 (67%) were principals 
and seven (33%) were assistant principals. Administrators reported that they attended 
relatively few sessions of LDC PLCs, with 35% attending less than one quarter, 15% attending 
about one quarter, and 20% attending about a half. Another 10% reported attending three 
quarters, and 20% attended more than three quarters. These numbers show some variation 
from teachers’ and project liaisons’ report of administrators’ attendance at meetings. 
Particularly, 35% of teachers reported that administrators attended more than three quarters 
of PLC meetings (in comparison to 20% as self-reported by administrators); it is possible, 
however, that teachers conflated other administrators (such as the district LDC project lead) 
with the principal or assistant principal who received this survey. 


45 


Training and Support (A3-4) 


Administrators had the opportunity to attend professional development sessions, either 
online or in person, such as Launch Day and administrator meetings. Over half (55%) 


attended two of these offerings. The range was 0-5, with an average of 1.85. 


Interactions with LDC were overwhelmingly positive. Almost all administrators agreed 
or strongly agreed that they were able to reach LDC staff with questions (95%); their school 
had adequate technology to access LDC (90%); LDC staff were able to connect them to 
additional resources (85%); LDC offered sufficient professional development for project 
liaisons (80%); and there were sufficient professional development opportunities for 


administrators (75%). 
Classroom Observation (A5-6) 


Almost half (45%) of administrators reported observing teachers implementing LDC 
three to five times during the year, while 40% observed 1 to 2 times, 10% observed six or 
more times, and 5% did not observe at all. These figures corroborate teachers’ reports of 
being infrequently observed, with 58% of teachers reporting that they were not observed at 
all during the school year, and 25% of teachers reporting one observation. In general, 


classroom observation was not a frequently used support structure. 


Almost all administrators (84%) who observed LDC instruction believed that LDC 


modules were moderately or very effective in developing students’ literacy skills. 
Impact on Teacher Practice (A7) 


A majority of administrators observed significant improvement in all areas of teacher 
practice probed on: 85% believed teachers had improved moderately or a great deal in 
selecting focus standards; 85% in identifying necessary skills to complete the writing 
assignment; 85% in creating standards-driven writing tasks; 75% in using evidence of 
student performance to shape instructional decisions; 70% in tracking and analyzing student 
progress in a systematic way; 70% in creating daily lessons to teach skills to complete the 
writing task; 70% in assessing students’ progress as they completed the writing task; and 


70% in identifying patterns of student understandings or misconceptions. 
Impact on Student Learning (A8) 


Administrators were almost universally positive about the effects of LDC on students. 
Large majorities of administrators agreed or strongly agreed that LDC helped students’ overall 
literacy performance (95%); LDC supported students’ college and career-ready skills (95%); 
LDC modules effectively supported students’ writing (90%); LDC improved students’ writing 
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quality (90%); LDC improved students’ content knowledge (85%); LDC helped students 
understand components of the writing task (85%); LDC helped students develop reading skills 
(80%); and LDC helped students develop speaking and listening skills (80%). 


Administrator Leadership Role (A9) 


Almost all administrators felt they played an active role in LDC implementation: 85% 
agreed or strongly agreed that they made changes to school schedules to accommodate LDC 
PLC time; 80% were able to shape LDC implementation at their schools; 72% met regularly 
with the LDC project liaison; 70% were involved in discussions about differentiating LDC 
implementation to meet teacher needs; and 60% led discussions about how to expand LDC 


implementation in future years. 
Alignment (A10) 


Administrators reflected on how well LDC aligned with other school initiatives, programs, 
and curricula (see Figure 3.7). Majorities of administrators agreed or strongly agreed that 
LDC was a strategy for implementing state college and career-ready standards (85%); LDC 
helped teachers create writing assignments to use within current curricula (85%); LDC 
complemented other initiatives at the school (80%); LDC helped students prepare for state 
assessments (65%); and that their school connected LDC to specific schoolwide goals (65%). 
Survey responses suggest that at least in some sites other initiatives being implemented 
simultaneously created a challenge, with about a third (35%) of administrators believing that 
the time spent on LDC interfered with other initiatives, and 70% reporting that it was difficult 
for teachers to focus on LDC because of other competing priorities. 
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It was difficult for teachers to focus on LDC 


because of other competing priorities at the 4+ —__—_ 7 
(n=20) 70% 


school. 


LDC helped prepare students in my school for 35% 
current state assessments. (n=20) 65% 
The time spent implementing LDC interfered 


with other important initiatives at my school. ——— 65% 
(n=20) 7 
LDC was a strategy for implementing my state’s 15% 
College- and Career-Ready Standards. (n=20) 85% 
LDC complemented other initiatives taking place 20% 
in my school. (n=20) 80% 
LDC helped teachers create writing assignments 15% 
to use within their current curricula. (n=20) 85% 
Our school connected LDC implementation to 35% 
our specific schoolwide goals. (n=20) 65% 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


O Disagree/Strongly Disagree @ Agree/Strongly agree 


Figure 3.7. LDC Alignment (A10) 


Scale-Up and Sustainability (A11) 


The outlook for LDC implementation in future years was moderately positive according 
to administrators, as reported in Figure 3.8. Over two thirds (70%) agreed or strongly agreed 
that teachers and administrators were committed to sustaining LDC; 70% expect most 
teachers to continue the following year; 55% expected the LDC PLC to increase in size; and 
40% of administrators observed new collaborations across grades and/or subjects. A small 
minority (15%) observed teachers who were not in LDC using the LDC planning process and 
tools. 
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Figure 3.8. Scale-up and Sustainability (A11) 


District Support (A12) 


Almost two thirds (65%) of administrators agreed or strongly agreed that district leaders 
supported the implementation of LDC; 55% reported that district leaders visited the school 
to discuss LDC implementation; 45% thought district professional development efforts were 
aligned with LDC; 45% agreed that district leaders were interested in expanding LDC to other 
schools; and 35% thought district leaders had a firm understanding of LDC. 


3.4 Open-Ended Responses for All Participants 


Each survey respondent had the opportunity to provide open-ended feedback in 
response to three questions: (1) What supports did you find the most useful and why?; (2) 
What supports were not helpful and why?; and (3) In what ways could LDC implementation 
be improved in your school in the future? 


Below are the summary findings across all three groups of respondents, including 
comments from 103 teachers, 18 project liaisons, and 18 administrators. Eighteen teachers 
(18%) reported they found all supports helpful, as did 11 project liaisons (61%), and 5 
administrators (28%). Six teachers (5%) commented that no part of the intervention was 
helpful. It is important to note that the majority of responses to all survey questions were 
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positive; with few exceptions, the specific concerns described below belong to a minority of 
respondents who felt strongly enough to comment in the open-ended response section. 


Professional Learning Community 


Weekly meetings with the LDC PLC received 20 positive comments, 18 of which were 
from teachers who nearly all commented that they appreciated the opportunity to collaborate 
with other teachers. 


The most common suggestion for improving LDC implementation in the future was to 
provide more protected time for the PLC. Time was mentioned at least 71 times, with related 
requests for more time and support to both plan and implement modules (not necessarily 
within the PLC time), to be paid for their time, and to use time more efficiently. Teachers 
were very positive about the content of the PLCs, and were most disappointed with the 
logistical issues (e.g., difficulty scheduling meetings, competing requirements, technology 
issues). AS one teacher put it, “The professional development part of the LDC was really 
powerful in that it gave us a protected hour per week in which we worked on lessons that 
mattered, that had scope and depth and were standards-based. I cherished that hour to 


|” 


improve my practice 


At least eight comments were made about needing an earlier start or more support to 
get the implementation off the ground. One teacher elaborated: “It would have been nice to 
have done this at the beginning of the year in order to start the year with full lessons ready 
to be implemented.” A handful of teachers specifically requested summer training and earlier 
start dates. A few teachers commented that they were given no choice about participation. 
One teacher would have appreciated “discussions prior to being implemented. I felt it was 
just thrown at me and I resent that.” Overall, there were nine comments, almost all from 
teachers, who felt that increased teacher buy-in was necessary in order for LDC to succeed 
at their sites. 


There were at least 24 comments about needing more institutional support. Notably, 10 
administrators (56%) requested more support from either the district or from LDC. Other 
requests for support were more specific to the PLC, such as expanded content in the CoreTools 
library. 


There were seven suggestions to add more teachers to PLCs by expanding across grade 


levels, subjects, or the entire school. 
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Coach (in-person) 


Here we report on comments about several types of in-person coaching, including virtual 
coaches’ site visits and assistance from the district liaison and LDC liaison. A few teachers 
also mentioned support from project liaisons in the context of coaching. There were 49 
comments indicating that in-person coaching was one of the most useful supports, and 31 
comments indicating that it was insufficient. These negative comments universally 
communicated the sentiment that the PLC did not receive enough or any in-person coaching. 
Several respondents said there was no substitute for in-person support. For at least a handful 
of respondents, in-person coaching was a necessary component of successful LDC 
implementation. 


Likewise, the second most common request among suggestions for improvement was 
to receive more in-person support, with 14 requests. Those that were more specific asked for 
help with seeing model lessons implemented in real time and in-person, individualized 


support. 


Some respondents applied their positive comments to both virtual and in-person 
coaching supports. One teacher differentiated the two in this way: “In-person coaching was 
useful because it helped me become more familiar with navigating through CoreTools and 
was able to get a better understanding of where I was headed and how to get there. It also 
gave me the opportunity to get immediate feedback. Virtual coaching helped lead, direct, and 
assist with maintaining focus.” 


Whether in person or virtually, teachers appreciated receiving feedback, which was 
mentioned 10 times. Teachers appreciated feedback in any format, whether via individual 
Zoom conferences, during PLC conferences, via email, or the CoreTools and Teaching Channel 


platforms. 
Coach (virtual) 


LDC coaches were the most frequently mentioned item, with 60 comments that they 
were one of the most useful supports and 28 comments that they were not helpful. These 
included comments about Zoom videoconference meetings with the entire PLC, individual 
Zoom meetings with coaches, email contact, and written feedback via CoreTools and Teaching 
Channel. 


Among the positive mentions, respondents praised coaches’ feedback and guidance in 
navigating LDC website content like CoreTools, the library of modules, and the online courses. 
They appreciated the ability to ask questions and receive immediate feedback. They also 
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appreciated one-on-one videoconferences with coaches and written feedback via CoreTools, 


Teaching Channel, and email. 


Respondents who felt virtual coaches were not helpful cited technical problems as the 
most common issue. Across all groups, there were 29 mentions of technology being a 
problem, and zero positive comments about technology. When the videoconferencing 
software and hardware did not function properly, participants felt there was time wasted. 
Other negative comments included that the virtual coaches were ineffective, with several 
teachers mentioning that a mid-year switch in coaches resulted in noticeable improvement. 
Several respondents just felt uncomfortable with the medium and preferred in-person, more 
traditional methods. Notably, among suggestions for improvement, 49 comments were made 
about requesting more in-person coach visits, while no participants requested more video 


conferences. 
CoreTools 


CoreTools received 60 positive comments, and 47 negative comments. Included in this 
count were comments about the content of the LDC library, most of which were positive. 
Although there were a large number of positive mentions, participants generally did not 
explain why they found CoreTools to be so useful. 


On the other hand, participants who struggled with CoreTools were very specific: there 
were dozens of complaints about the user interface and how difficult it was to use the site. 
The three most commonly cited barriers to mastering use of CoreTools were ease of use, lack 
of time to explore, and lack of appropriate modules. A few science teachers were unhappy to 
discover that there were very few resources available to them. Three of them were upset that 
they were creating free curricular content for the LDC library and reaping none of the library’s 
benefits. One teacher said, “Right now I can find lessons easier on Google. This gives me less 
incentive to use LDC because I feel like I am only contributing but I am not receiving any 


benefits when it comes to the science modules.” 
Online courses 


Online courses received only two mentions (both positive) in the survey, indicating that 


this support was not widely used among these participants. 
Teaching Channel 


Teaching Channel resources, including videos and discussions, were mentioned five 
times altogether: one positive mention and four negative mentions. As with online courses, it 


did not appear from the comments that this was widely used resource. 
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3.5 Summary of Results 


This survey captured the responses of 78% of all participants in the district’s first year 


of implementation. In addition to answering closed-ended questions, most of the respondents 


also provided narrative comments about aspects of the program they felt were useful as well 


as those that could use improvement. Generally, respondents provided positive feedback. 


Overall, the survey results suggest the following: 


Across all participant groups, survey responses showed positive attitudes toward 
LDC. All measures of satisfaction or improvement were rated positively by more 
than half of respondents. 


Three quarters of teachers and 95% of administrators agreed that LDC helped 
improve students’ literacy performance. 


LDC coaches received almost unanimous positive feedback, with 95% of teachers 
reporting that their coaches gave them appropriate and timely feedback and 
support. A small number of participants submitted negative comments regarding 
coaches. 


Project liaisons were almost universally reported to be highly approachable, 
effective, and knowledgeable. Almost all teachers reported that their 
administrators encouraged LDC participation at the school, though only 47% of 
teachers reported receiving feedback from administrators about LDC and 58% 
reported never being observed while teaching an LDC task. 


Over two thirds of teachers felt they had sufficient time to meet during professional 
learning community (PLC) sessions and that administrators protected that time. 
However, the most frequently requested modification for future years was for more 
protected time during and outside of PLCs to plan modules, implement, and receive 
feedback about implementation. Most teachers reported meeting once every two 
weeks, which was about half the recommended frequency of PLC meetings. 


About two thirds of teachers expressed interest in learning more about how to lead 
LDC implementation at their schools. Over half of project liaisons and 
administrators expected their teachers to continue with LDC the following year. 
Both groups observed that teachers at the school who did not participate in the 
LDC PLC generally did not adopt LDC methods or materials on their own. 


While 77% of teachers agreed that their school had adequate technology to support 
LDC implementation, the second most common complaint by participants (19%) 
was the difficulty in successfully conducting Zoom meetings. 


With regard to creating modules, teachers were most likely to report success with 
selecting focus standards, creating the writing assignment, identifying skills, and 
providing multiple opportunities to engage students. They were less likely to report 
success with selecting texts, making connections to previous or future work, 
creating daily lessons, and planning a variety of assignments. 


With regard to implementing LDC in the classroom, teachers were most likely to 
report success with engaging students in productive struggle, differentiating 
instruction, and locating evidence of progress in student work. They were slightly 
less likely to report success in using evidence to modify plans, providing feedback, 
and systematically collecting information about student learning. 
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e When asked about LDC impact on students, teachers reported positive impact on 
writing quality, college and career readiness skills, overall literacy performance, 
reading skills, and content knowledge. 

This year’s survey responses were consistently positive across all dimensions and 
participant groups. Suggestions and recommendations provided in this report were primarily 
about adapting to and ameliorating district constraints, such as protected time for meetings 
and technological limitations. 
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4.0 Analyses of LDC CoreTools Data 


In the following section, we report on how participants interacted with LDC’s CoreTools 
online system. We begin in Section 4.1 by presenting participation rates for key CoreTools 
activities, including creating a user account, viewing modules and mini-tasks, editing modules 
and mini-tasks, and commenting on modules and mini-tasks. In Section 4.2, we dig deeper 
into CoreTools viewing, editing, and commenting by sharing descriptive statistics for all i3 
CoreTools users followed by results broken down by key subgroups (including LDC role, school 
level, and content area taught). In Section 4.3, we compare the level of engagement for i3 
CoreTools users who completed and taught a full length module to those users who did not 
complete and teach a module during the school year. 


4.1 CoreTools Activity Participation Rates 


The four key CoreTools activities we examined are: creating a user account, viewing 
modules and mini-tasks, editing modules and mini-tasks, and commenting on modules and 
mini-tasks. Among the 188 CoreTools users, 154 identified were teachers, and 34 of them 
were administrators, with some individuals in each category acting as project liaisons. As seen 
in Table 4.1, nearly all participants used CoreTools to at least some degree. Ninety-one (91) 
percent of all participants created a user account, 84% of participants viewed modules or 
mini-tasks, 76% edited modules or mini-tasks, and 47% of them commented on modules or 
mini-tasks. 
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Table 4.1 


CoreTools Feature Participation Rates: 2016-17 


Participant Type 


N of 
Participants 


N and % of 
Participants 
with User 
Accounts 


N and % of 
Participants 
who viewed 
a module or 
mini-task 


N and % of 

Participants 

who edited a 
module or 
mini-task 


N and % of 
Participants 
who 
commented 
on a module 
or mini-task 


Teacher 


Teacher/Project 
Liaison 


Total Teachers 


Project Liaison* 
Administrator 


Administrator/Project 
Liaison 


Total Administrators* 


Total Participants 


139 
15 


34 


188 


127 (91%) 
14 (93%) 


141 (92%) 
5 (100%) 
23 (85%) 


2 (100%) 


30 (88%) 


171 (91%) 


120 (86%) 
14 (93%) 


134 (87%) 
5 (100%) 
15 (56%) 


2 (100%) 


22 (65%) 


156 (84%) 


107 (77%) 
14 (93%) 


121 (79%) 
4 (80%) 
16 (59%) 


1 (50%) 


21 (62%) 


142 (76%) 


37 (27%) 
5 (33%) 


42 (27%) 
2 (40%) 
2 (7%) 


0 (0%) 


A (12%) 


AG (24%) 


*We categorize the coaches and coordinators who completed the Project Liaison survey as 


administrators. 


In addition to displaying participation rates on key CoreTools activities for all i3 


participants, Table 4.1 also displays subgroup results for participants playing different roles 


in the i3 implementation. These subgroups mirror the five groups to whom we administered 


surveys: teachers, 


project 


liaisons, 


administrators, 


teacher/project 


liaisons, and 


administrator/project liaisons. Teachers typically had higher participation rates than the 


administrators, as expected. 


Over 90% of participants created a CoreTools user account. Significantly, even 


administrators who did not play a project liaison role created user accounts in high numbers. 


This suggests that the vast majority of administrators overseeing LDC PLCs had at least some 


familiarity with the online platform. 


Likewise, large majorities of PLC teachers and project liaisons (between 87% and 


100% depending on the participant role) viewed modules and/or mini-tasks in CoreTools. 


Administrators who did not play a project liaison role were considerably less likely to engage 


in this way, although a majority still viewed at least one module or mini-task. 
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We consider editing modules and mini-tasks to be the key indicator of deep engagement 
with the CoreTools module building platform. Nearly four out of every five participating 
teachers edited at least one module or mini-task. Administrators were less likely than teachers 


to edit materials in CoreTools; nevertheless, over half of administrators did so. 


Overall, participants were much less likely to add comments to modules or mini-tasks 
(one quarter did so). This perhaps should not be surprising, as the i3 model set an expectation 
that coaches provide feedback to teachers via comments, but did not require teachers to 


comment back or reflect on peers’ work. 


4.2 Engagement with Key CoreTools Activities 


In this section, we describe participants’ level of engagement with three key CoreTools 
activities: viewing modules and mini-tasks, editing modules and mini-tasks, and commenting 
on modules and mini-tasks. Descriptive statistics are reported for all participants, as well as 
a number of subgroups. Those subgroups capture the role the individual played in LDC 
(teacher, project liaison, and administrator), the level of the school at which the participant 
works (elementary, middle, or high), and in the case of teachers, the content area taught 
(elementary/multiple subjects, secondary ELA, secondary history/social studies, secondary 
science, and special education). As noted earlier, some participants played multiple roles in 
the intervention, so the teacher, project liaison, and administrators groups overlap to some 
degree as they do in our survey analysis. Descriptive statistics are only reported for groups 
with 4 or more members. As a result, we do not report on certain groups, such as math 
teachers. Finally, note that for each of the three analyses below, participants who did not 
engage in the activity are excluded from the analysis (i.e., the analyses do not include any 
observations with zero views, edits, or comments). As such, in each case, the results describe 
the behavior of participants who engaged in the particular activity at least once. The samples 
for the viewing, editing, and commenting analyses are 156, 142, and 46 respectively (see 
bottom row of Table 4.1) 


Module/Mini-Task viewing. In Table 4.2, we present descriptive statistics on how 
many times i3 participants viewed modules and mini-tasks, both overall and by subgroup. We 
present the minimum number of views, the maximum number of views, the mean number of 
views, the standard deviation, and the sum total views across all participants. Overall the 
average participant viewed modules or mini-tasks about 30 times, while the range was from 
one view to 136 views. The standard deviation of 27.1 suggests that about two thirds of all 


participants viewed modules or mini-tasks between 3 and 57 times. 
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Table 4.2 


Descriptive Statistics for the Number of Times a Participant Viewed a Module or Mini-Task in 
CoreTools, by Participant Subgroup 


Subgroup N Min Max Mean Sar abh Sum 

All Participants 156 1 136 29.9 27.1 4661 
Participant Role 

All Teachers 134 2 136 31.0 25.9 4147 

All Project Liaisons 21 4 125 48.0 32.4 1008 

All Administrators 17 1 8211.5 20.0 195 
School Level 

Elementary School Level BB A «136>. 3325 30.0 2951 

Participants 

Middle School Level Participants 44 4 99 21.7 18.1 953 

High School Level Participants 24 2 125 31.5 27.7 757 
Content Area Taught 

Taught Elementary/Multiple 69 3 136 = 36.7 30.2 2535 

Subjects 

Taught Secondary Science 130 5 = =62 28.5 18.5 370 

Taught Secondary ELA 19 4 60 26.7 18.4 507 

Taught Secondary Social 12. 7 99° 26,2 23.3 497 

Studies/History 

Taught Special Education 9 2 38 15.4 12.3 139 


As would be expected, project liaisons on average had the greatest number of views, 
while administrators viewed the least number of times. There was considerable variation in 
viewing behaviors across participant subgroups within these categories. Participants in middle 
schools viewed somewhat fewer modules and mini-tasks than their elementary and high 
school peers. Interestingly, at the secondary level, ELA, history/social studies, and science 
teachers all viewed similar numbers of modules and mini-tasks on average. Special education 


teachers seemed to view fewer modules or mini-tasks than their regular education peers. 


Module/ Mini-Task editing. On average, participants who engaged in editing modules 
or mini-tasks, did so 19 times over the course of the school year. There was a wide range of 
engagement from editing once to making 81 edits to modules and mini-tasks. Two thirds of 


participants edited between three and 35 times. Table 4.3 reports the descriptive results. 
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Table 4.3 


Descriptive Statistics for the Number of Times a Participant Edited a Module or Mini-Task in 
CoreTools, by Participant Subgroup 


Subgroup N Min Max Mean rire Sum 
All Participants 142 1 81 19.1 16.2 2712 
Participant Role 
All Teachers 121 1 81 19.9 15.7 2405 
All Project Liaisons 19 1 81 25.8 20.8 491 
All Administrators 17 2 65 10.2 14.9 174 
School Level 
Elementary School Level 80 «1 81 17.6 17.0 1407 
Participants 
Middle School Level Participants 39 1 47 17.7 12.5 689 
High School Level Participants 23. 1 63 26.8 17.0 616 
Content Area Taught 
Taught Elementary/Multiple 62 1 81 19.1 17.3 1182 
Subjects 
Taught Secondary Science 130 1 47 21.2 16.7 275 
Taught Secondary ELA 17 1 52 24.5 15.0 417 
Taught Secondary Social 15 7 42 25; 11.6 377 
Studies/History 
Taught Special Education 7 1 30 11.0 10.1 77 


As with page viewing results, project liaisons edited the most and administrators edited 
the least. Participants at the high school level showed greater engagement in the editing 
process than their elementary and middle school peers, at least as measured by the number 
of edits. ELA and social studies teachers edited more documents than their elementary and 
science teacher peers, while special education teachers were less likely to make edits. Despite 


these differences, there was great variation across participants within each of the subgroups. 


Module/Mini-Task commenting. Only 47 participants commented on modules or 
mini-tasks in the past school year. Within that highly engaged group, participants commented 
between one and 26 times, and on average four times. Most commenters were teachers, but 
those administrators who did comment on average did so more often than teachers or project 


liaisons. Similar to the results for editing, high school participants engaged to a greater degree 
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than elementary and middle school participants on average. Differences in means across 


content areas were small. 


Table 4.4 


Descriptive Statistics for the Number of Times a Participant Commented on a Module or Mini- 


Task in CoreTools, by Participant Subgroup 


Subgroup N Min Max Mean pei dee Sum 

All Participants 46 1 26 4.1 4.8 190 
Participant Role 

All Teachers 42 1 16 3.7 3.6 157 

All Project Liaisons 1 26 4.7 9.4 33 

All Administrators 4 1 26 8.3 12.0 33 
School Level 

Elementary School Level 26 1 16 3.5 3.8 91 

Participants 

Middle School Level Participants 11 1 9 3.4 2.7 37 

High School Level Participants 9 1 26 6.9 7.9 62 
Content Area Taught 

Taught Elementary/Multiple 22 1 16 3.7 4.1 81 

Subjects 

Taught Secondary Science 1 5 3.2 1.6 16 

Taught Secondary ELA 1 12 4.1 4.0 29 

Taught Secondary Social 1 9 4.0 3.3 20 


Studies/History 


4.3 CoreTools Engagement as an Implementation 


Variable 


To evaluate the validity of CoreTools engagement as an indicator of LDC implementation, 


we examined the relationship between the three CoreTools engagement measures and 


module implementation. As described in the next Chapter, CRESST identified full length 


modules that we were confident i3 participants created and/or adapted and taught in their 


classrooms. We took the presence of uploaded student work as evidence of teachers having 


taught the module in their classrooms. This parameter yielded a sample of 53 modules, 


associated with 50 teachers and 2 administrators. Figure 4.1 displays the mean number of 


CoreTools views, edits, and comments for the 50 teachers who completed and taught full- 
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length LDC modules and their 104 peers for whom we do not have evidence of full module 
implementation. As shown, teachers who completed and taught full-length LDC modules 
exhibited over twice as much viewing, editing, and commenting activity than their peers. The 
results suggest that participants who engage deeply with the module building platform are 
more likely to complete and teach LDC modules. 


45.0 
40.0 
35.0 
30.0 


25.0 


42.7 
24.3 
19.3 

20.0 
= 11.4 
10.0 

5.0 

1.8 06 


Number of CoreTools Views Number of CoreTools Edits Number of CoreTools Comments 


Teachers who completed and taught modules (n=50) 


Teachers who did not complete and teach modules (n=104) 


Figure 4.1. Mean number of CoreTools views, edits, and comments made by teachers who did and did 
not complete and teach modules 
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5.0 Module Artifact Analysis 


Table 5.1 presents mean scores and standard deviations for all modules analyzed. 
Among the 53 modules we scored, 33 were at the elementary level, and 20 of them were at 
the secondary level. As previously stated, each module was rated using a five-point scale to 
measure 1) the effectiveness of the writing task or objective(s) for the module, 2) the 
module’s alignment to the standards, 3) fidelity to the four stages of LDC instructional practice 
(i.e., preparation for the task, reading process, transition to writing, and writing process), 4) 
the quality of the instructional strategies, 5) coherence and clarity of the module, and 6) the 
overall quality of the module (see Appendix A for full descriptions of the dimensions). 


Table 5.1 
Module Means by Dimension and School Level 


Elementary Secondary All modules 
Dimension n M (SD) n M (SD) n M (SD) 
1. Effective writing task 33, 3.6(1.2) 20 3.1 (1.1) 53 3.4 (1.2) 
2. Standards alignment 3300 3.41.1) 20 2.9 (1.3) 53 3.0 (1.2) 


3. Fidelity to LDC module 33 3.6(1.3) 20 2.2 (1.6) 53 3.0 (1.6) 
instruction 

4. Quality instructional 33 3.1(1.3) 20 1.9 (1.4) 53 2.6 (1.4) 
strategies 

5. Coherence and clarity of 33 3-1(1.3) 20 2.2 (1.5) 53 2.7 (1.4) 
module 

6. Overall impression 33 -3.0(1.2) 202.1 (1.5) 53 2.6 (1.4) 


As can be seen, modules at the elementary level received higher mean ratings than did 
the modules at the secondary level. More specifically, mean ratings for the elementary 
modules ranged from 3.0 to 3.6, indicating that on average all dimensions were moderately 
present or realized. In contrast, only Dimensions 1 and 2, which focus on the effective writing 
task and standards alignment were in the three range for the secondary modules. In most 
cases, modules submitted for use at the middle and/or high school level received ratings of 


near two, indicating that the dimensions were not present or barely present or realized. 


For the remainder of this report, we present results for the elementary and secondary 
modules separately. This will include further descriptive results, results from generalizability 
theory analyses, as well as factor analyses for the elementary modules. Because of the small 


sample size for the secondary modules, reported findings here will be limited to descriptive 
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results for the content area subgroups; generalizability and factor analyses for the secondary 
sample will be conducted at a later time once a larger sample of eligible modules has been 


collected. 
5.1 Elementary Module Results 


Table 5.2 presents descriptive results for the elementary modules by content area. A 
few comparative observations can be made. First, when looking across content areas, 
modules tended to receive the highest ratings for Dimension 1, concerning the effective 
writing task, and Dimension 3, which focused on fidelity to LDC module instruction. Second, 
the dimensions where modules received somewhat lower means varied depending upon the 
content area. More specifically, only science modules received a mean lower than three in 
regard to Dimensions 4 and 6, which focus on the quality of instructional strategies and the 
raters’ overall impression of the module. In contrast, social studies teachers seemed to 
struggle with Dimension 2, concerning standards alignment. Ratings seem to indicate that 


teachers had issues with specifying appropriate focus standards for their modules. 


Table 5.2 
Descriptive Statistics for the Elementary Modules by Content Area 


ELA Science Social Studies 

Dimension n M (SD) n M (SD) n M (SD) 

1. Effective writing task 13 3.5 (1.0) 14 3.4 (1.5) 5 4.0 (0.0) 

2. Standards alignment 13° 3.3 (1.0) 14 3.1 (1.2) 5 2.6 (1.3) 

3. Fidelity to LDC module 13 3.5(1.5) 14 3.5 (1.5) 5 3.6 (0.9) 
instruction 

4. Quality instructional 13 3.2(1.5) 14 2.9 (1.2) 5 3.6 (0.9) 
strategies 


5. Coherence andclarityof 13 3.2(1.4) 14 3.1 (1.3) 5 34 (1.3) 
module 


Generalizability theory. As previously noted, two models were used to examine 
potential error in the scoring process. Table 5.3 and 
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Table 5.4 present results from the two-faceted and one-faceted models, which examine 
error across and within dimensions. As would be the goal of any rating session, most of the 
variation can be attributed either directly to the differences in the modules created by the 
teachers (66%) or to differing quality across the modules by dimension (28%). We were also 
encouraged to see that less than one percent of the variation was due to the raters either 


directly or across dimensions. 


Table 5.3 
Generalizability Study of the Elementary Module Ratings across Dimensions 


Source of Error Var. % 
Module (o2t) 1.06 66.4 
Rater (o2r) 0.00 0.0 
Dimension (02d) 0.06 3.8 
Module * Dimension (o2td) 0.45 27.8 
Rater * Dimension (o2rd) 0.00 0.1 
Module * Rater (o2tr) -0.00 0.0 
Error (o2trd,e) 0.03 1.9 


Note. Negative estimates of variance were changed to zero in order to calculate 
percentages (see Shavelson & Webb, 1991) 


As previously mentioned, we also used a one-faceted design to further disentangle the 
very small amount of variance that we found due to the dimensions (4%). In this case, more 
than 99% of the variation in most of the dimensions could be attributed directly to the 
modules created by the teachers. The only exception involved Dimension 4, concerning the 
quality of instructional strategies, where 11% of the variance was due to an interaction 
between raters and modules. This indicates that anchor points and/or training for Dimension 


4 might warrant revisions before the next year’s rating sessions are conducted. 
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Table 5.4 
Generalizability Studies of the Elementary Module Ratings for Each Dimension 


Module (o2t) Rater (o2r) Module * Rater — Error (o2trd,e) 
(o2tr) 

Var. % Var. % Var. % Var. % 
1. Effective 1.31 100.0 0.00 0.0 -0.01 0.0 0.00 0.0 
writing task 
2. Standards 1.32 100.0 0.00 0.0 -0.01 0.0 0.00 0.0 
alignment 
3. Fidelity to 1.71 99.6 -0.00 0.0 0.01 0.4 0.00 0.0 
LDC module 
instruction 
4. Quality 1.46 88.7 0.01 0.5 0.18 10.8 0.00 0.0 
instructional 
strategies 
5. Coherence 1.84 99.8 0.00 0.0 -0.00 0.2 0.00 0.0 
and clarity of 
module 
6. Overall 1.43 100.0 0.00 0.0 -0.00 0.0 0.00 0.0 
impression 


Note. Negative estimates of variance were changed to zero in order to calculate percentages (see 
Shavelson & Webb, 1991) 


Factor analysis. Table 5.5 presents the principal component solutions extracted from 
the teacher scores for each dimension across content areas. The result of the factor analysis 
is encouraging in that for the elementary sample, all six dimensions loaded on one factor and 
accounted for 74.56% of the variance in ratings across the 33 modules. This suggests a 
dominant factor or trait underlying the six dimensions used for this measure of LDC 
implementation. 
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Table 5.5 


Principal Component Analysis of the Elementary Module Ratings 


Dimension 


Elementary (n = 33) 


. Effective writing task 

. Standards alignment 

. Fidelity to LDC module instruction 
. Quality instructional strategies 


. Coherence and clarity of module 


nu BR WHN 


. Overall impression 


82 
.69 
.87 
92 
.88 
.97 


5.2 Secondary Module Results 


Table 5.6 presents descriptive results for the 19 secondary modules by content area. As 


the content area subgroups are quite small, please be cautious in interpreting the results in 


this section. Based on Table 4.6, modules tended to receive higher ratings for Dimensions 1 


and 2, which focus on the writing task and the alignment to state content standards, with 


mean ratings above or approaching three. This indicates that these two dimensions were often 


moderately present or realized. Secondly, means were generally quite low for Dimensions 3 


through 6, indicating that these traits were on average barely or not present at all. This seems 


to be in line with comments made by raters during debriefings, who noted that the potential 


of the module, as seen in “Section 1: What Task,” was not always carried through the 


remainder of the module. Because of the sample size limitation, we did not run either the 


generalizability theory analyses or the factor analysis, as we did for the elementary sample. 


Table 5.6 


Descriptive Statistics for the Secondary Modules by Content Area 


ELA Science Social Studies 

Dimension n M (SD) n M (SD) n M (SD) 

1. Effective writing task 5 3.0 (0.7) 5 3.4 (1.5) 9 2.8 (1.0) 

2. Standards alignment 5 3.0 (1.6) 5 3.0 (1.4) 9 2.6 (1.0) 

3. Fidelity to LDC module > 2.6 (2.1) 5 2.2 (2.2) 9 1.8 (1.1) 
instruction 

4. Quality instructional 5 1.8 (1.6) 5 2.0 (2.0) 9 1.8 (1.1) 
strategies 

5. Coherence and clarity 5 1.6 (1.5) 5 2.2 (1.9) 9 2.3 (1.3) 
of module 
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5.3 Qualitative Results 


Expert raters were asked to participate in debriefings in an attempt to shed light on their 
ratings and suggest potential ways to improve ratings in the future. These debriefings were 
conducted at the end of each set of rating sessions (e.g. Elementary, ELA, etc.). The following 
presents key findings concerning Dimension 1 through 5, as well as our expert raters’ overall 


impressions. 


Dimension 1: effective writing task. While this was often the highest scoring 
category, few teachers took full advantage of the opportunity to explain their writing task. 
Many teachers simply filled in mandatory sections of the teaching task, without providing the 
optional questions allowed at the beginning. Furthermore, many provided limited information 
or none at all in the optional student background section. Our expert raters felt strongly that 
this additional information was necessary to know what was expected of students and to 


determine what the expectations were for higher order thinking. 


Dimension 2: standards alignment. Teachers were inconsistent in their specification 
of standards for the modules. First, some modules included standards primarily in the 
designated area in the “What Task” section of the module creator, other modules included 
them with the mini-tasks, and yet others included information on standards in more than one 
location. This inconsistency caused raters to spend more time trying to disentangle issues of 
alignment. It also should be noted that raters felt that teachers often lacked focus in specifying 
their standards, with some providing too many and others providing too few. In some cases, 
this was hypothesized to be an artifact of teachers adapting rather than completing original 


modules. 


Dimension 3: fidelity to LDC module instruction. While ratings concerning fidelity 
to LDC module instruction varied greatly in quality, especially among the secondary modules, 
raters generally felt that teachers were more successful in the early rather than the later parts 
of the instruction. Students were given less instruction than considered necessary for the 
transition to writing and writing process skills. Furthermore, teachers used default mini-tasks 
that showed little adaptation or that did not align with the teaching task. 


Dimension 4: quality instructional strategies. Feedback on the quality of 
instructional strategies tended to mimic what was discussed concerning Dimension 3. Raters 
believed that artifacts showed little or no evidence of how teachers were scaffolding critical 
thinking. Many of the modules used the same mini tasks, the writing task as specified often 
did not match the task template or the student work samples, and many modules felt like 
they were “cut and pasted” together. 
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Dimension 5: coherence and clarity of module. As one of our raters noted, 
coherence and clarity of the modules was “almost always either great or terrible.” Raters 
noted a disconnect between the instructional strategies and what students produced. 
Furthermore, they considered it difficult to rate this dimension in cases where teachers 
provided little student work. In the end, though, modules that raters felt that they could take 
and implement directly in their own classrooms tended to receive the highest ratings for this 


dimension. 


Dimension 6: overall impressions. Original modules were typically thought to be of 
much higher quality than those that were adapted. These modules were easier to understand, 
seemed to be more thought out, and had fewer mismatches across the task sections. Raters 
also felt that because they lacked understanding of what actually was changed or customized 
in the adapted modules, they had a limited ability to judge the quality of the work. Our expert 
teachers suggested that if more teachers had completed the student background and teacher 
reflection sections, and had included more student work samples, this would have greatly 
helped in the rating process. In addition, raters suggested that the addition of a section to 
provide details about the students, such as information about English language learners or 
prior achievement, would have been helpful in determining the appropriateness of the task 


template and mini tasks. 


5.4 Summary of Results 


While the overall sample and content area subsamples for the elementary module 
ratings were fairly small, and should therefore be interpreted with caution, some fairly clear 
trends emerged. First, teachers at the elementary level were moderately successful in the 
development of their modules as evidenced by the mean ratings for the dimensions that were 
primarily in the low- to mid-three range. This was generally true for the overall elementary 
sample, as well as the three content area subsamples. Second, results of the Generalizability 
study provided evidence that raters did a good job, with most variation due directly to 
differences in quality across the modules (66.44%) or to differing quality across the modules 
by dimension (27.78%). Likewise, Generalizability study results showed that almost all 
variation in ratings for the individual dimensions were due to differences in the modules 
(88.66% to 100.00%). The factor analysis provides further evidence that our scoring captured 
true variation in quality across modules. All six dimensions loaded on one factor, which 


accounted for 74.56% of the variance in ratings across the elementary modules. 
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In contrast, analyses for the secondary modules were limited because of having only 20 
modules overall and less than 10 per content area subsample. Despite these limitations, it 
does appear that secondary teachers were most successful in regards to their specification of 
an effective writing task and the alignment to content standards, areas which generally 
received ratings in the three range. In contrast, these teachers were generally less successful 
regarding their LDC fidelity, instructional strategies, and coherence and clarity. These trends 
should still be considered tentative and will be examined further once we have a larger sample 


of secondary modules to analyze in the coming year. 
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6.0 Student Outcome Analysis 


This section presents the student outcome analysis we conducted to evaluate LDC’s 
impact on student learning in the 2016-17 school year. We begin by describing the process 
we used to define the LDC student sample and to construct a matching comparison sample. 
We then present descriptive statistics for the treatment and comparison groups. Finally, we 
report the estimated impact of LDC on students as measured by Smarter Balanced scores in 
English Language Arts (ELA). 


6.1 LDC Sample and the Matching Process 


As described earlier, our LDC teacher sample included both elementary and middle 
school teachers in the study district. Separate sampling and analyses were conducted for 
these two groups of teachers, as described in Section 2.5 of this report (Analytical 
Approaches). 


Elementary sample. The eligible LDC student sample for elementary includes all 
students who were enrolled for the entire 2016-17 school year under the instruction of at 
least one of the participating LDC teachers, and who had available data both for prior and 
current year achievement scores and demographic characteristics. Achievement and 


demographic data were used in the matching process. 


The comparison sample was selected via a multi-stage process. In the first stage, we 
identified comparison schools for each of the participating LDC schools, based on similarity in 
both prior achievement and demographics. Then from within these comparison schools, we 
selected comparison students who were similar to each of the LDC students. We used the 
following variables to identify similar comparison schools: the percentage of students eligible 
for free or reduced price lunch, the percentage of African American students, mean prior 
student achievement in ELA, mean prior student achievement in math, the attendance rate 
of teachers, the percentage of teachers with three or fewer years of teaching experience, and 
the school grade span. At the student-level, we used coarsened exact matching (CEM) and 
matched students exactly on grade level as well as their Hispanic, Gifted, and English 
Language Learner (ELL) status. We also matched closely on variables for Black, Asian, other 
ethnicity, free or reduced priced lunch status, special education, and on prior individual 
student achievement and the mean prior achievement of students’ core content class peers. 


Finally, students also were matched on the average attendance rate of their core content area 
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teachers and the percentage of their core content teachers? with three or fewer years of 


teaching experience, within five percentage points across the two samples. 


The 2016-17 elementary sample prior to the CEM process included 13 schools, the 32 
elementary teachers who consented to participate in the evaluation study, and their 794 
students. After the student-level matching our final elementary LDC sample included 711 


elementary students and retained the same number of teachers and schools (see Table 6.1). 


Prior to matching, the potential elementary comparison sample consisted of 563 schools, 
3,847 teachers, and 87,521 students. This comparison sample was substantially reduced 
during the first stage of matching, which identified the five schools that most closely 
resembled each of the 13 LDC elementary schools. After student-level CEM, a workable 
analytic comparison sample consisted of 42 schools, 168 teachers and 711 students (see 
Table 6.1). 


Table 6.1 
Before and After Matching Sample Sizes: Elementary School Analysis 


LDC sample Comparison sample 
School Teacher Student School Teacher Student 
Stage 1 
Before matching 13 32 794 563 3,847 87,521 
Stage 2 
After matching 13 32 711 42 168 711 


Middle school sample. The eligible LDC student sample for middle school was more 
challenging to identify than the elementary school sample because students were taught by 
multiple teachers of different subjects, each of whom did or did not participate in LDC. Our 
first step in sampling was to define at each school the potential sample of LDC students. These 
were all students at the school who enrolled in and completed two semesters of a standard 
ELA course. We began by classifying ELA courses as either standard core content ELA or 
supplemental ELA courses (for example, a second ELD class intended to support students’ 


3 In the vast majority of cases, each elementary student had only one teacher and so the 
prior experience variable and teaching experience variables reflected that one teacher. 
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success in the standard grade level course). Almost all of these students also were enrolled 
and completed core class content in social studies/history, and science in 2016-17. To be 
identified for the LDC sample, a student needed to have taught by at least one LDC teacher 
in ELA, social studies/history or science. In addition, to be part of the sample, prior and current 
achievement scores needed to be available for each student. The same two stage process 
used for selecting the elementary comparison sample was used for the middle school 
sampling. 


The resulting LDC sample included six schools, 35 teachers, and 2,450 students prior to 
the CEM process. After the CEM student-level matching, our final secondary LDC sample was 
reduced to 2,200 students (see Table 6.2). 


Table 6.2 
Before and After Matching Sample Sizes: Middle School Analysis 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 6 35 2,450 178 6,597 91,981 
After matching 6 35 2,450 30 1,995 19,499 
Stage 2 
After matching 6 35 2,200 29 1,056 2,200 


Prior to matching, the potential comparison sample consisted of 178 schools, 6,597 
teachers and 91,981 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified the five schools that most closely resembled each of 
the six LDC schools. After student-level CEM, a workable analytic comparison sample of 29 
schools, 1,056 teachers, and 2,200 students was constructed (see Table 6.2). 


6.2 Descriptive Results on the Matched Analytic 
Samples 


Tables 6.3 and 6.4 present the student characteristics for the LDC student and 
comparison students, based on the final analytical samples for the elementary and secondary 


analyses respectively. Treatment and comparison samples matched very closely. Exact 
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matching was achieved on some variables, and all demographic variables were within 5 
percentage points. We used the spring 2017 Smarter Balanced ELA tests as our outcome 
measures, and the spring 2016 Smarter Balanced ELA tests as one of the matching variables. 
For the prior achievement matching variable, we standardized Smarter Balanced scale scores 
at each grade level relative to district performance, based on the district mean and standard 
deviation for the ELA test at each grade level. Standardizing scores in this way enables us to 
easily compare our samples’ performance relative to the districts and to compare scores 
across grades and years more easily. A standardized scale score of zero, for example, 
indicates that the student scored at the mean for all other students in the district who took 
the same test. A standardized scale score of 1.0 meant that the student scored one standard 
deviation higher than the district mean. Conversely, a standardized scale score of -1.0 


indicated that the student scored one standard deviation lower than the district mean. 


The final LDC elementary student sample after matching was comprised largely of 
Hispanic and Black students with more than two-thirds of them being of Hispanic ethnicity 
while slightly over one-quarter were Black (see Table 6.3). The sample was also mostly 
comprised of 4 and 5‘ grade students (95.4% combined), and of students with low 
socioeconomic status (88.5%). English language learners represented about one-quarter of 
this sample, while special education students represented slightly over one quarter of the 
sample, and less than 10% were classified as gifted students. Mean performance on the prior 
year academic assessment was very close to the district-wide performance levels in math and 
ELA. 


As shown in Table 6.4 the final LDC secondary student sample included a large 
proportion of Hispanic students (87.5%), and a similarly large proportion of students with low 
socioeconomic backgrounds (88.5%). The secondary students were required to have received 
two full semesters of core ELA content, as is reflected in Table 6.4. We also captured student’s 
exposure to supplemental ELA content and, on average, this was slightly less than one-half 
semester (0.4 supplemental units) per student. Core social studies and science content 
exposure was also typically two full semesters per student. The sample was mostly comprised 
of 6 (16.1%), 7% (41.1%), and 8" grade students (42.8%). English language learners 
represented slightly less than one-quarter of this sample, while there were about twice as 
many gifted students (16.5%) as compared to special education students (8.2%). In addition, 
mean performance on the prior year academic assessment was somewhat lower for LDC 


students as compared to district-wide performance levels in math and ELA. 
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Table 6.3 


Student Characteristics of Elementary School Treatment and Comparison Groups After 
Coarsened Exact Matching, 2016-17 School Year 


Student characteristic LDC sample: Comparison 
Elementary sample: 
(n = 711) Elementary 
(n = 711) 

Race/Ethnicity 

Hispanic (%) 68.8 68.8 

Black (%) 26.7 25.9 

Asian (%) 2.5 2.8 

White (%) 1.1 1.0 

Other (%) 0.9 1.5 

Female (%) 50.2 49.4 
Special programs status 

Poverty (%) 88.5 90.4 

English Language Learner (%) 26.2 26.2 

Special education (%) 29.6 28.6 

Gifted (%) 6.6 6.6 
Student prior achievement 

Mean prior year math Z score 0.06 0.03 

Mean prior year ELA Z score 0.05 0.04 
Class & teacher characteristics 

Mean peer prior ELA Z score of current peers 0.05 0.05 

Mean teacher attendance (%) 97.0 96.8 

ae teachers with <3 years of experience 6.7 10.7 
Grade level 

Grade 6 in 2016-17 (%) 41.5 41.5 

Grade 7 in 2016-17 (%) 53.9 53.9 


Grade 8 in 2016-17 (%) 4.6 4.6 


Table 6.4 


Student Characteristics of Middle School Treatment and Comparison Groups After Coarsened 
Exact Matching, 2016-17 School Year 


Student characteristic LDC sample: Comparison 
Secondary sample: Secondary 
(n = 2,200) (n = 2,200) 

Race/Ethnicity 

Hispanic (%) 87.5 85.4 

Black (%) 5D or 

Asian (%) 22 1.4 

White (%) 4.0 5a 

Other (%) 0.8 2.5 
Female (%) 51.1 49.2 
Special programs status 

Poverty (%) 89.3 88.2 

English Language Learner (%) 22.9 22.9 

Special education (%) 8.2 9.1 

Gifted (%) 16.5 12.5 
Student prior achievement 

Mean prior year math Z score -0.14 -0.14 

Mean prior year ELA Z score -0.13 -0.14 
Class & teacher characteristics 

Mean peer prior ELA Z score of current peers -0.10 -0.16 

Mean teacher attendance (%) 95.9 96.5 

Percent teachers with <3 years of experience 

(%) 5.9 8.2 
Number of content units 

ELA core content units 2.0 2.0 

ELA supplemental units 0.4 0.4 

Social studies units 2.0 2.0 

Science units 2.0 2.0 
Grade level 

Grade 6 in 2016-17 (%) 16.1 16.1 

Grade 7 in 2016-17 (%) 41.1 41.1 

Grade 8 in 2016-17 (%) 42.8 42.8 
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6.3 Outcome Analysis Results: Elementary Sample 


Because elementary students occasionally were taught by multiple teachers rather than 
a single one, we also employed a Multi-Membership Multiple Classification (MMMC) design for 
the elementary analysis. Two different approaches were used to model the LDC treatment 
intervention variable as a fixed effect at the student-level. The first model was dosage 
dependent, which took into account some variation in elementary students’ level of exposure 
to LDC teachers. In this approach the treatment variable was structured as a continuous 
response variable between zero and one, based on exposure to LDC teachers for ELA, social 
studies/history and/or science. In contrast, the second approach was modeled as dosage 
independent and considers any student exposed to any LDC intervention teacher via at least 
one subject as a treated individual. In this latter approach the treatment variable was 
dichotomous, coded as one for LDC treated students and zero for comparison students. The 
outcome variable for both models was students’ Smarter Balanced scores in English language 
arts. For technical reasons of evaluating effect sizes, Smarter Balanced scale scores were 


standardized to the study sample. 
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Table 6.5 


2016-17 LDC Elementary School Effect Estimates on Smarter Balanced ELA 
Performance Dosage Dependent Model 


Variables Model coefficient (S.E.) 
Level 2 - LDC teacher treatment -0.032 (0.071) 
Level 1 - Student characteristics 
Hispanic -0.075 (0.088) 
Black -0.203 (0.089)* 
Other ethnicity -0.032 (0.159) 
Poverty 0.007 (0.051) 
Female 0.129 (0.031)* 
English Language Learner -0.191 (0.046)* 
Special education -0.096 (0.056) 
Gifted 0.200 (0.066)* 
Grade 5 0.082 (0.054) 
Grade 6 -0.089 (0.157) 
Prior year math Z score 0.261 (0.027)* 
Prior year ELA Z score 0.582 (0.026)* 


Note. Most treated students were only exposed to one teacher (the LDC intervention teacher), 
thus the average treated student in this sample received a 0.91 treatment dosage. Because of 
this, we could estimate an average treatment on the treated (ATT) at 0.91 * -0.032 = -0.029. 
*Significant at p =.05 


The dosage dependent effect of having an LDC teacher on elementary students’ ELA 
performance in 2016-17 is shown in Table 6.5, and the dosage independent effect is shown 
in Table 6.6. Most treated students were only exposed to one teacher (the LDC intervention 
teacher); thus the average treated student in this sample received a 0.91 treatment dosage. 
Model results show no statistically discernible LDC effect on student outcomes. Similarly, the 
dosage independent model revealed no statistically significant effect for LDC. In other words, 
both analyses showed that students taught by LDC teachers scored similarly on the ELA test 


to their matched peers in the comparison group. 


The effects of the covariates on student performance also were similar under both 
models and were in the expected directions. Prior ELA performance was the strongest 
predictor and prior math performance also helped explain the outcome. In addition to prior 
achievement, four demographic variables helped predict performance. English language 


learners performed at lower levels than English Only and Reclassified Fluent English Proficient 
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(RFEP) students, black students performed at lower levels on average than White and Asian 
students, females performed at significantly higher levels than males, and gifted students 
also performed at higher levels than their peers. 


Table 6.6 


2016-17 LDC Elementary School Effect Estimates on Smarter Balanced ELA 
Performance Dosage Independent Model 


Variables Model coefficient (S.E.) 
Level 2 - LDC teacher treatment -0.001 (0.066) 
Level 1 - Student characteristics 
Hispanic -0.075 (0.088) 
Black -0.203 (0.089)* 
Other ethnicity -0.032 (0.159) 
Poverty 0.007 (0.051) 
Female 0.129 (0.031)* 
English Language Learner -0.192 (0.046)* 
Special education -0.096 (0.056) 
Gifted 0.200 (0.066)* 
Grade 5 0.083 (0.054) 
Grade 6 -0.096 (0.156) 
Prior year math Z score 0.261 (0.027)* 
Prior year ELA Z score 0.582 (0.026)* 


Note. *Significant at p =.05 


6.4 Outcome Analysis Results: Middle School 
Sample 


Similar analyses were conducted at the middle school level, where the norm was student 
exposure to multiple teachers. As discussed in detail in Chapter 2, students could be exposed 
to anywhere between O and 6 different LDC teachers. Using a Multiple Membership Multiple 
Classification (MMMC) design, each observation at Level 1 represented one student, which 
was linked to the ELA, social studies/history and science teachers that student was exposed 
to during the year. Weights across teachers for each student summed to a unity (1). 


As with the elementary sample, we present results of models that are both dosage 
dependent and dosage independent. The dosage dependent effect of being exposed to LDC 
teachers in ELA, social studies, and science on middle school students’ ELA performance in 
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2016-17 is shown in Table 6.7, and the dosage independent effect of having at least one LDC 
teacher is shown in Table 6.8. Model results show no statistically discernible LDC effect on 
student outcomes for either model. In other words, both analyses showed that students 
taught by LDC teachers scored similarly on the ELA test to their matched peers in the 
comparison group. It should be noted that while we initially included the number of 
supplemental ELA courses that middle school students completed as a covariate, results for 


this variable were not significant and so were not included in the final middle school models. 


The effects of the covariates on student performance were also similar to those in the 
elementary school model. Prior ELA and math performance as well as ELL status, Black 
ethnicity, and gender were significant predictors of ELA performance. In secondary schools, 
Hispanic ethnicity and the average prior achievement of a student’s peers were also significant 
predictors of ELA performance in the expected directions. 
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Table 6.7 


2016-17 LDC Middle School Effect Estimates on Smarter Balanced ELA Performance 
Dosage Dependent Model 


Variables Model coefficient (S.E.) 
LDC teacher treatment 0.103 (0.102) 
Student characteristics 
Hispanic -0.204 (0.041) * 
Black -0.194 (0.052) * 
Other ethnicity -0.015 (0.077) 
Poverty 0.003 (0.029) 
Female 0.153 (0.017) * 
English Language Learner -0.147 (0.028) * 
Special education -0.062 (0.034) 
Gifted 0.055 (0.029) 
Grade 7 0.076 (0.049) 
Grade 8 0.016 (0.052) 
Mean teacher attendance rate 0.028 (0.007) * 
tees teachers with <3 years of 0.161 (0.099) 
Mean peer prior ELA Z score 0.153 (0.022) * 
Prior year math Z score 0.263 (0.015) * 
Prior year ELA Z score 0.551 (0.016) * 


Note. Since the average treatment student received a 0.446 treatment dosage we could 
estimate an average treatment on the treated (ATET) at 0.446 * 0.103 = 0.046. This effect 
was not statistically significant. 

*Significant at p =0.05 


Table 6.8 


2016-17 LDC Middle School Effect Estimates on Smarter Balanced ELA Performance 
Dosage Independent Model 


Variables Model coefficient (S.E.) 
LDC teacher treatment 0.033 (0.060) 
Student characteristics 
Hispanic -0.204 (0.041) * 
Black 20.195 (0,052) * 
Other ethnicity 0.018 (0.076) 
Poverty 0.003 (0.029) 
Female 0.153 (0.016) * 
English Language Learner -0.149 (0.028) * 
Special education -0.062 (0.034) 
Gifted 0.052 (0.029) 
Grade 7 0.075 (0.049) 
Grade 8 0.016 (0.052) 
Mean teacher attendance rate 0.028 (0.007) * 
ee teachers with <3 years of 0.163 (0.099) 
Mean peer prior ELA Z score 0.152 (0.022) * 
Prior year math Z score 0.263 (0.015) * 
Prior year ELA Z score 0.551 (0.016) * 


Note. *Significant at p =0.05 


6.5 Summary of Results 


Quasi-experimental design analysis of the impact of LDC on 2016-17 student test scores 
in elementary and secondary schools produced two measures of impact effect at each school 
level. One impact effect measure took into account the dosage of LDC exposure while the 
other reflected any exposure to an LDC teacher and was thus dosage independent. Our results 
for 2016-17 did not reveal any statistically significant intervention effects. This finding should 
not be surprising given the early stage of intervention, with teachers having only completed 


one year of implementation. 
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7.0 Summary of Findings 


This annual report presents an initial look at LDC implementation in the first cohort of 
20 schools in a large West Coast district during their first year of implementation. We 
summarize these results organized by the three categories of evaluation questions we listed 
in Chapter 1 of this report. 


7.1 Program Characteristics and Implementation 


All participant groups in this district were generally positive about LDC characteristics 
and implementation at their sites. Questions about program quality and impact received 
positive responses from a majority of respondents. A quarter of survey respondents took the 
time to specify in open-ended responses that they found all LDC supports helpful. 


Teachers valued the collaborative nature of the initiative, as evidenced by their 
participation in regular PLC meetings and informal discussions. A large majority of teachers 
credited LDC with making them more likely to collaborate with other teachers, not only in 
their grade levels and content areas but outside of them as well. In addition to peer feedback, 
teachers valued coach feedback greatly. Positive ratings of LDC coaches were nearly 


universal. 


Online course material was revised sufficiently following pilot teachers’ feedback that it 
became a strength of the program this year. Most teachers rated the online courses as highly 
useful, helpful, and relevant, though a small majority did not find them easy to use. 


There were 154 teachers over multiple content areas across all grade levels, and some 
of these teachers did not feel well supported by the available resources, particularly in science 
and math. These teachers did less well on instructional module quality, engaged with the 
CoreTools module building platform to a lesser degree, and a few of them detailed their 
criticisms about the fit between their teaching contexts and the program’s offerings in open- 


ended responses. 


Module quality suffered when teachers simply adapted existing modules without making 
significant changes or providing insight about their reasoning for instructional decisions. 
Although the module building platform provides optional areas where teachers provide such 
reasoning and reflection after the fact, most teachers did not utilize these areas. In addition, 
few teachers provided student work. Without either student work or teacher reflection, it was 


difficult for raters to evaluate whether or not instructional goals were met. Teachers who 
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implemented modules and provided evidence of the implementation engaged more deeply 
with the LDC platform. 


7.2 Contextual Factors and Implementation 


The people who supported teachers in the LDC implementation — project liaisons, 
administrators, LDC coaches, and LDC staff - received extremely positive responses for 
helpfulness, timeliness, and level of knowledge. Questions about the quality of support that 
participants received often had universal or nearly universal agreement. The effectiveness of 


the people involved was the most notable facilitating factor in implementation. 


Time was one of the primary hindering factors in implementation. Although most 
teachers reported that they thought they had sufficient time for their LDC PLCs to meet, they 
did not all meet with the recommended frequency or duration. On the other hand, a small 
majority of teachers reported in open-ended responses that time was problematic. The open- 
ended responses revealed that teachers needed different types of support when it comes to 
time. Some teachers needed their administrators to protect that time. Others requested more 
time to write, implement, and reflect. Some teachers want a different use of the time, whether 
it is to get more support with the online resources, to observe others in the classroom, or to 


meet with a specific grade level or content area team rather than as a large group. 


Related to time was the problem of technology. This issue had two points. First, the 
logistics of Zoom meetings were difficult at some sites. This was perceived as a frustrating 
waste of time. These open-ended responses stood in contrast to the finding that most teachers 
thought their school’s technology was adequate for LDC implementation. This indicates that 
although the Zoom meetings worked for the majority of participants, failed meetings resulted 
in highly negative perceptions for those affected. The second problem with technology was 
that some teachers needed far less time and support to understand CoreTools and other 
teachers need far more. In general, however, the complexity of CoreTools and the need to 
use multiple sites like The Teaching Channel was a significant barrier for those teachers who 
were affected. Thus, the most effective use of limited teacher and coach time during PLC 
meetings were constrained by the different technology needs of the sites and of the 
participating teachers. 


The issue of differentiation among teachers touches on many factors in addition to 
technology, such as content area as described above, and as the initiative progresses, the 


level of experience and knowledge about the LDC model. As the pool of teachers both deepens 
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and widens, the ability of the program to differentiate will likely affect participants’ experience 
to a great extent. 


7.3 Program Impacts 


Based on survey results, LDC was perceived to have positive impacts on both teacher 
practice and student learning. Administrators were particularly enthusiastic, with a large 
majority believing that LDC helped students in multiple facets of literacy and skills acquisition. 
Teachers were slightly more cautious but still positive. 


Teacher self-reports of change in their own ability over the year indicated that they 
felt they had improved most in areas related to instructional planning, which aligned with the 
primary content of LDC sessions. Slightly fewer teachers reported positive change in areas 
involving collecting and using student data to inform instruction, which again corresponded 
to the fewer number of teachers who implemented modules. 


Module analysis showed strengths in creating an effective writing task and fidelity to 
LDC module instruction. Science modules scored less well on quality of instructional strategies 
and overall impression of the rater. History/social studies scored less well on standards 
alignment. 


Our preliminary analysis of the 2016-17 student data did not reveal any statistically 
significant intervention effects. This finding should not be surprising given the early stage of 
intervention, with teachers having only completed one year of implementation while LDC 
implementation is a two-year process. 
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Appendix A: LDC Module Rating Dimensions 


Each module was rated on six dimensions. All of these were rated using a five-point scale. With the first 
five, anchor points ranged from not present or realized to fully present or realized. In contrast, the overall 
dimension ranged from inadequate to advanced. 


Dimension 


1 Effective Writing Task 


Definition 

Degree to which teaching task makes effective use of the template task’s writing mode (i.e., argumentation 
or explanation); requires sustained writing and effective use of ideas and evidence to substantiate claims; 
and is feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information 
Module Creator Handout (Task) 


¢ Read and evaluate the teaching task, student background/prior knowledge, and summary information. 
¢ Evaluate the difficulty or ease students may encounter trying to answer the question. 


* Compare module teaching task to teaching task template options. 


Anchor Points Description 


5 Fully Present or Realized The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order 
thinking and writing, and are appropriate for the grade-level and 
subject matter. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized | Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level 
and subject matter. 


2 Barely Present or Realized 


1 NotPresent or Realized There is minimal evidence of an effort to identify an explicit and 
clear teaching task and performance expectations that provide 
opportunities for students to think critically and are appropriate for 
the grade-level and subject matter. 
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Dimension 


2 


Alignment to CCSS, Local, State Literacy and Content Standards 


Definition 


Extent to which module addresses content essential to the discipline, as well as reading comprehension 
and writing standards informed by local and state standards. 


Main Sources of Information 


Module Creator Handout (Task) 


¢ Read and evaluate the standards included in the module. 


* Module should include ELA as well as subject matter CCSS/state standards. 


* Compare and contrast the standards the module includes with those that could have been included. 


¢ Particular attention to content standards (CCSS History/Social Studies, Science, and Technical 
Subjects); State Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 


Anchor Points 


Description 


5 Fully Present or Realized 


Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the 
topic and teaching task. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


2 Barely Present or Realized 


1 Not Present or Realized 


Module broadly addresses content essential to CCSS and local or 
state standards in science or social studies and reading 
comprehension and writing. Standards are sufficiently aligned to the 
topic and teaching task. 


Minimal evidence that module addresses content essential to the 
discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 
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Dimension 


3 


Fidelity to LDC Module Instruction 


Definition 
Degree to which module instruction, activities, and teaching task address each of the four stages of 
instructional practice (preparation for the task, reading process, transition to writing, writing process). 


Main Sources of Information 
Module Creator Handout (Instruction) and Information Sheet 


* Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 


3 Moderately Present or Realized | The module instruction, activities, and teaching task reflect 


instruction. 


Anchor Points Description 


5 Fully Present or Realized The module instruction, activities, and teaching task reflect 
deliberate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect demonstrable effort 
to develop instructional scaffolding within and across each stage of 
instruction. 


4 Sufficiently Present or Realized 


moderate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect sufficient effort to 
develop instructional scaffolding within and across each stage of 
instruction. 


2 Barely Present or Realized 


1 NotPresent or Realized The module instruction, activities, and teaching task reflect poor 
attention and lack of fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect inadequate effort to 
develop instructional scaffolding within and across each stage of 
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Dimension 


4 


Quality Instructional Strategies 


Definition 


Degree to which the module provides clear instructional strategies aimed at helping students develop 


literacy skills and successfully complete the teaching task. In addition, the degree to which module 


instruction and activities scaffold critical thinking and performance in a way that is meaningful within the 


context of a given field or subject matter. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


¢ Evaluate extent to which the module activities scaffold critical thinking and student performance within 


the context of the subject matter at the core of the teaching task. 


e¢ Evaluate extent to which instructional strategies guide student learning in literacy and ability to 


complete the teaching task. 


Anchor Points 


5 Fully Present or Realized 


4 Sufficiently Present or Realized 


SSS eee — eeea—ses<—V3s—w—e—S—eeooeeooooooooaoewanmm—maannmnmnnwnnwmwnmnmwm¢5n<_—qva eee ee ae 


Description 


Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking 
in social studies or science. There is explicit attention to helping 
students develop an accurate understanding of the topic and 
teaching task, and literacy skills necessary to complete the writing 
task successfully. 


3 Moderately Present or Realized 


2 Barely Present or Realized 


Instructional strategies and activities are available to support 
adequate student learning and critical thinking in social studies or 
science. There is moderate attention to helping students develop an 
understanding of the topic and teaching task, and literacy skills 
necessary to complete the writing task. 


1 NotPresent or Realized 


Limited instructional strategies and activities are available to support 
student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding 
of the topic and teaching task, or literacy skills necessary to 
complete the writing task. 
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Dimension 


3 


Coherence and Clarity of Module 


Definition 


The degree of logical alignment found between the teaching task and the goals of the module with the 
readings, mini-tasks, and instructional strategies. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


Anchor Points 


Description 


5 Fully Present or Realized 


Strong alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


2 Barely Present or Realized 


1 Not Present or Realized 


Moderate alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


Poor alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 
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Dimension 


S) 


Overall Impression 


Definition 
A holistic assessment of the LDC Module. 


Main Sources of Information 
Module Creator Handout, Classroom Handouts, and Student Work 


* To what extent does this module contribute to student college readiness and development of advanced 
literacy skills? 


Anchor Points 


5 Advanced LDC Module Implementation 


4 Proficient LDC Module Implementation 


3 Adequate LDC Module Implementation 


2 Marginal LDC Module Implementation 


1 Inadequate LDC Module Implementation 
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Appendix B: 2016-2017 Teacher Survey 
and Responses 


T1. 


T2a. 


T2b. 


T2c. 


T2d. 


T3. 


LDC Participation 


Please select your school from the drop down box. 
Teachers are skipped to T3 if they teach in an elementary school. 


In the current school year (2016-17), how many classes did you teach? 
[_] classes N = 52, Mean = 5.17, Range: 0-8 


In how many of these classes did you use LDC modules and/or mini-tasks? 
[_] classes N = 52, Mean = 2.90, Range: 0-7 


In what content areas did you use LDC modules and/or mini-tasks? 


In what grades did you use LDC modules and/or mini-tasks? 


Prior to the current school year (2016-17), did you have any experience with 
LDC? 
(N = 122) 


Yes 10 teachers (8.2%) 
No @ SkiptoT5a 112 teachers (91.8%) 


T4. How many of the following did you teach prior to the current school year 
(2016-17)? 


1.8, Range: 0-6 


[__] LDC modules N = 10, Mean 


LDC mini-tasks, outside of 


N = 10, Mean 
modules 


2.2, Range: 0-6 
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Professional Learning Community and Teacher 
Collaboration 


T5a. Did you participate this year in a Professional Learning Community (PLC) at 
least partly focused on implementing LDC in your school? 


(N = 110) 
Yes Skip toT6 104 teachers (94.5%) 
No 6 teachers (5.4%) 


T5b. Did you use any LDC tools in your instructional planning or classroom 
instruction this year? 


(N = 6) 
Yes Skip to 5d 6 teachers (100%) 
No 0 teachers (0%) 


T5c. Why did you choose not to use any LDC tools in your instructional planning or 
classroom instruction this year? 


[Survey ends here for respondents answering question T5c] 


T5d. What LDC tools did you use during the current school year? Select all that 
apply. 
(N = 6) 


4 teachers (66.7%) 


CoreTools online platform to access existing modules 
or mini-tasks 


CoreTools online platform to design modules or mini- 4 teachers (66.7%) 


tasks 

LDC online courses 1 teachers (14.3%) 
Modules or mini-tasks given to me by other teachers 1 teachers (14.3%) 
in my school 


Other (please specify) 


O teachers (0%) 


[Survey ends here for respondents answering question T5d] 
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T6. About how often did your LDC PLC meet? 


Less than once a month 
Once a month 
Every other week 


Once a week 


Twice a week or more often y Skip to T8 


(N = 116) 


0 teachers (0%) 

28 teachers (24.1%) 

53 teachers (45.7%) 
@ Skip toT8 29 teachers (25%) 


6 teachers (5.2%) 


T7. What were the primary barriers preventing your LDC PLC from meeting 


weekly? Select all that apply. 


support. 


PLC time was not protected. 
PLC members had limited interest in attending meetings. 
School administrator did not make it a priority. 

Project liaison did not provide sufficient organizational 


Not enough teachers participated. 


PLC members had other priorities that compete with LDC 
participation. 


Other (please specify) 


(N = 81) 

22 teachers (27.2%) 
13 teachers (16%) 
10 teachers (12.3%) 


3 teachers (3.7%) 
7 teachers (8.6%) 


48 teachers (59.3%) 


28 teachers (34.6%) 


T8.About how often did you have informal discussions (as opposed to scheduled 


meetings) about LDC with teachers in your LDC PLC? 


Once a month 

Every other week 
Once a week 

Twice a week or more 


(N = 116) 


Less than once a month 21 teachers (18.1%) 


20 teachers (17.2%) 
37 teachers (31.9%) 
27 teachers (23.3%) 
11 teachers (9.5%) 


T9.On average, how long did your school’s LDC PLC meetings typically last? 


Less than 45 minutes 
45 to 59 minutes 

60 to 74 minutes 

75 minutes or more 


(N = 116) 

6 teachers (5.2%) 
84 teachers (72.4%) 
25 teachers (21.6%) 
1 teachers (0.9%) 
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LDC Training and Support 


T10. How effective was your LDC PLC in the following areas? 


Not A little Moderately Very 

effective effective effective effective 
Demonstrating the 
usefulness of LDC for 4 (3.4%) 26 (22.4%) 60 (51.7%) 26 (22.4%) 
teachers (N = 116) 
Creating an environment in 
which teachers were é é a j 
comfortable working 3 (2.6%) 17 (14.7%) 47 (40.5%) 49 (42.2%) 
together (N = 116) 
Fostering an environment 
where teachers shared their a a ' . 
instructional plans with 4 (3.4%) 26 (22.4%) 42 (36.2%) 44 (37.9%) 
colleagues (N = 116) 
Allowing space for shared a ' ij 5 
problem solving (N = 116) 4 (3.4%) 31 (26.7%) 41 (35.3%) 40 (34.5%) 
PI Te acetone ato 6 (5.2%) | 31 (26.7%) | 50 (43.1%) | 29 (25.0%) 


develop modules (N = 116) 
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T11. How would you rate each of the following aspects of the online course 


material (in the Learn tab in LDC CoreTools) that your coach used or directed 


you to use? 


Poor Fair Good Excellent 
Clarity of information 
presented 10 (8.6%) 30 (25.9%) 56 (48.3%) 20 (17.2%) 
(N = 116) 
Relevance of information 
presented 9 (7.8%) 28 (24.1%) 55 (47.4%) 24 (20.7%) 
(N = 116) 


Ease of use (N = 116) 


18 (15.5%) 


43 (37.1%) 


44 (37.9%) 


11 (9.5%) 


Usefulness of resource 
documents (e.g., LDC 
Pitfall Checklist, CCSS 
Mental Markers, etc.) 

(N = 116) 


19 (16.4%) 


32 (27.6%) 


51 (44.0%) 


14 (12.1%) 


Usefulness of videos 
(N = 116) 


20 (17.2%) 


36 (31%) 


46 (39.7%) 


14 (12.1%) 


Degree to which course 
material helped me to 
create and/or adapt LDC 
modules (N = 116) 


13 (11.2%) 


44 (37.9%) 


45 (38.8%) 


14 (12.1%) 


Opportunity to extend 
learning when needed or 
desired (N = 116) 


16 (13.8%) 


32 (27.6%) 


50 (43.1%) 


18 (15.5%) 


T12a.Overall, were you able to get the feedback and support you needed from 
your LDC coach (through written feedback in LDC CoreTools, or coaching and 
modeling in your LDC PLCs) to plan, teach, reflect on, and revise LDC 


modules)? 


(N = 116) 
Yes 105 teachers (90.5%) 
No 11 teachers (9.5%) 


T12b.Did your LDC coach provide written feedback on your module(s) in LDC 
CoreTools in a timely manner? 


(N = 116) 
Yes 101 teachers (87.1%) 
No 15 teachers (12.9%) 
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T13. Outside of the PLC meetings with your LDC coach, please indicate whether 
you used each of the following types of coach support, and how helpful you 
found these types of support. 


Used 


Did not 
use 


Not 
helpful 


A little 
helpful 


Moderately 
helpful 


Very 
helpful 


Written feedback in 
LDC CoreTools 
from your LDC 
coach (in the 
comments 

areas and/or via 
the teacher 

work rubric) 

(N = 116) 


24 (20.7%) 


9 (7.8%) 


11 (9.5%) 


A1 (35.3%) 


31 (26.7%) 


One-on-one Zoom 
video conference 
and/or call with 
your LDC coach 
(N = 115) 


25 (21.7%) 


9 (7.8%) 


12 (10.4%) 


27 (23.5%) 


42 (36.5%) 


Coach-facilitated 
discussions on the 
Teaching Channel 
Teams platform 
(N = 116) 


40 (34.5%) 


7 (6.0%) 


19 (16.4%) 


28 (24.1%) 


22 (19.0%) 


Email or phone 
communication 
with your LDC 
coach 

(N = 116) 


37 (31.9%) 


7 (6.0%) 


13 (11.2%) 


29 (25.0%) 


30 (25.9%) 


Other (please 
specify) (N = 27) 


20 (74.1%) 


1 (3.7%) 


2 (7.4%) 


2 (7.4%) 


2 (7.4%) 
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Module Creation 


714. During the current school year (2016-17), how many LDC modules did you 
individually or collaboratively adapt from existing modules (e.g., modules you 
created in a prior year and/or modules found in the LDC Library in 
CoreTools)? 


[ ] Adapted modules N =104, Mean = 1.60, Range: 0-12 


T15. During the current school year (2016-17), how many LDC modules did you 
create, either individually or with colleague(s)? Only include modules built 
from scratch, not those adapted from existing modules in the LDC library. 


[__] New modules N = 104, Mean = 1.14, Range: 0-4 


T16. How did members of your PLC collaborate to create LDC modules? 
Check all that apply. 


(N = 122) 
Modules were created by individual teachers. 44 teachers (35.8%) 
ean were created by teams of two or more 62 teachers (50.4%) 
eachers. 
Modules were created by the PLC as a whole. 28 teachers (22.8%) 
Other (please specify) 12 teachers (9.8%) 
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T17. Please indicate to what extent you were able to do each of the following 
when creating LDC modules and/or mini-tasks. 


Not at all 


A little bit 


A 
moderate 
extent 


A great 
extent 


Select a set of focus 
standards for a writing 
assignment (N = 116) 


4 (3.4%) 


10 (8.6%) 


47 (40.5%) 


55 (47.4%) 


Create a standards- 
driven writing assignment 
task (N = 116) 


5 (4.3%) 


15 (12.9%) 


52 (44.8%) 


44 (37.9%) 


Select high quality, 
complex texts and other 
materials that allowed 
students to engage in 
deeper learning 

(N = 116) 


6 (5.2%) 


25 (21.6%) 


56 (48.3%) 


29 (25.0%) 


Create a writing 
assignment that provided 
multiple opportunities for 
students to engage with 
the material (N = 116) 


10 (8.6%) 


17 (14.7%) 


53 (45.7%) 


36 (31.0%) 


Identify the skills 
students need to develop 
in order to complete a 
writing assignment 

(N = 116) 


7 (6.0%) 


18 (15.5%) 


60 (51.7%) 


31 (26.7%) 


Create daily lessons to 
teach the skills a student 
needs to complete a 
writing assignment 

(N = 116) 


10 (8.6%) 


31 (26.7%) 


51 (44.0%) 


24 (20.7%) 


Plan for a variety of 
methods to assess 
student progress (e.g., 
mini-task scoring guides) 
(N = 116) 


10 (8.6%) 


35 (30.2%) 


54 (46.6%) 


17 (14.7%) 


Make connections to 
previous or future 
learning that make a 
writing assignment 
relevant for students 
(N = 116) 


10 (8.6%) 


27 (23.3%) 


53 (45.7%) 


26 (22.4%) 
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Classroom Implementation 


T18. How many total LDC modules did you teach during the current school year 
(2016-17)? 


[ Modules 


T19. Outside of modules, approximately how many individual LDC mini-tasks did you 
teach during the current school year (2016-17)? 


[_] Mini-tasks N = 116, Mean = 2.59, Range: 0-20 


N = 116, Mean = 1.67, Range: 0-6 


T20. Please indicate to what extent you were able to do each of the following 


activities when teaching LDC modules and/or mini-tasks. 


Not at all 


A little bit 


A moderate 
extent 


A great 
extent 


Engage students in 
productive struggle 
through challenging 
texts, discussion 
questions, and other 
instructional supports 
(N = 116) 


7 (6.0%) 


19 (16.4%) 


59 (50.9%) 


31 (26.7%) 


Provide support for 
students to complete the 
assignment by 
differentiating 
instruction (N = 116) 


8 (6.9%) 


19 (16.4%) 


60 (51.7%) 


29 (25.0%) 


Systematically collect 
information about 
student progress (N = 
116) 


11 (9.5%) 


30 (25.9%) 


49 (42.2%) 


26 (22.4%) 


Provide feedback to 
students using shared 
expectations for learning 
(e.g., a mini-task 
scoring guide) 

(N = 116) 


11 (9.5%) 


26 (22.4%) 


51 (44.0%) 


28 (24.1%) 


Locate evidence of 
performance of 
standards in student 
work (N = 116) 


8 (6.9%) 


25 (21.6%) 


55 (47.4%) 


28 (24.1%) 


Use evidence of student 
performance of 
standards to modify 
instructional plans 

(N = 116) 


9 (7.8%) 


27 (23.3%) 


54 (46.6%) 


26 (22.4%) 


100 


T21. 


T22. 


T23. 


T24. 


T25. 


T26. 


Did you teach a One-Text One-Week module? 
(N = 116) 
Yes 46 teachers (39.7%) 


No @ SkiptoT24 70 teachers (60.3%) 


What was the name of the One-Text One-Week module? 


Which of these statements best describes how you created the One-Text One- 
Week module? 
(N = 46) 
I created a module from a template in 28 teachers (60.9%) 
CoreTools. 
I copied and adapted another teacher’s 18 teachers (39.1%) 


module from the LDC Library in CoreTools. 


What module did you work on most heavily during the current school year 
(2016-17)? This module is typically one you worked on after the One-Text 
One-Week module. 


Which of these statements best describes how you created the module named 
in the previous question? 


(N = 116) 
I created a module from a template in 55 teachers (47.4%) 
CoreTools. 
I copied and adapted another teacher's 61 teachers (52.6%) 
module from the LDC Library in CoreTools. 
Did you teach this module in your classroom? 
(N = 116) 
Yes, I have already taught this module this year. 79 teachers (68.1%) 


No, but I plan to teach this module before the end of the 17 teachers (14.7%) 
2016-17 school year. 


No, but I plan to teach this module during next school 16 teachers (13.8%) 
year. 


No. I do not currently have plans to teach this module in 4 teachers (3.4%) 
my classroom. 
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Module Peer Review 


T27. How many modules did you submit online for LDC National Peer Review 


during the current school year (2016-17)? 


[| Modules If none, y skip to T29 N = 116, Mean = 0.70, Range: 0-4 


T28. How helpful did you find the National Peer Review process in improving the 


quality of your module? 


Not helpful 
A little helpful 
Moderately helpful 


Very helpful 


(N = 54) 

12 teachers (22.2%) 
19 teachers (35.2%) 
19 teachers (35.2%) 
4 teachers (7.4%) 
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7 


Impact on Teacher Practice and Learning 


T29. Between the beginning and end of this year’s work with LDC, please indicate 
how much your ability in the following areas has improved: 


Not at all 


A little 


Moderately 


A great deal 


Selecting a set of focus 
standards for a writing 
assignment (N = 116) 


10 (8.6%) 


18 (15.5%) 


59 (50.9%) 


29 (25%) 


Creating standards-driven 
writing tasks of sufficient 
quality and scope to allow 
students to think deeply 
about disciplinary content and 
practice literacy skills 

(N = 116) 


9 (7.8%) 


25 (21.6%) 


56 (48.3%) 


26 (22.4%) 


Identifying the skills students 
need to develop in order to 
complete a writing 
assignment (N = 116) 


10 (8.6%) 


17 (14.7%) 


63 (54.3%) 


26 (22.4%) 


Creating daily lessons to 
teach each skill a student 
needs to complete a writing 
assignment (N = 116) 


13 (11.2%) 


32 (27.6%) 


52(44.8%) 


19 (16.4%) 


Assessing students’ progress 
as they work toward 
completing a writing 
assignment (N = 116) 


13 (11.2%) 


26 (22.4%) 


57 (49.1%) 


20 (17.2%) 


Tracking and analyzing 
evidence about student 
progress in a systematic way 
(N = 116) 


12 (10.3%) 


33 (28.4%) 


54 (46.6%) 


17 (14.7%) 


Identifying patterns of 
student understandings or 
misconceptions (N = 116) 


11 (9.5%) 


36 (31%) 


53 (45.7%) 


16 (13.8%) 


Using evidence of student 
performance of standards to 
shape future instructional 
decisions (N = 116) 


12 (10.3%) 


27 (23.3%) 


54 (46.6%) 


23 (19.8%) 
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T30. Please indicate the degree to which you agree or disagree with the statements 


below. 
Strongly F Strongly 
disagree pisegtce agree agree 
Participating in LDC raised 
my expectations for 10 (8.6%) 17 (14.7%) | 72 (62.1%) | 17 (14.7%) 


students’ writing. 
(N = 116) 


Using LDC modules became 
an important part of my 
instructional practice. 

(N = 116) 


13 (11.2%) 


35 (30.2%) 


56 (48.3%) 


12 (10.3%) 


Implementing LDC helped 
me incorporate my state’s 
College- and Career-Ready 
Standards into my 
instruction. 

(N = 116) 


10 (8.6%) 


27 (23.3%) 


66 (56.9%) 


13 (11.2%) 


LDC helped me incorporate 
writing assignments into 
my existing curriculum. 

(N = 116) 


10 (8.6%) 


16 (13.8%) 


73 (62.9%) 


17 (14.7%) 


I am more likely to 
collaborate with other 
teachers on designing 
instruction after 
participating in our LDC 
Professional Learning 
Community. (N = 116) 


11 (9.5%) 


16 (13.8%) 


69 (59.5%) 


20 (17.2%) 


Participating in LDC helped 
me develop working 
relationships with teachers 
in different grades and/or 
subjects. (N = 116) 


9 (7.8%) 


27 (23.3%) 


63 (54.3%) 


17 (14.7%) 


I often incorporated LDC 
mini-tasks into my non- 
LDC instruction. (N = 116) 


14 (12.1%) 


46 (39.7%) 


43 (37.1%) 


13 (11.2%) 
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Impact on Student Learning 


T31. Please indicate the degree to which you agree or disagree with the statements 


below. 
Strongly F Strongly 
disagree Disaduce Agree agree 
LDC helped students develop ‘ ‘ ‘ ‘i 
reading skills. (N = 104) 10 (9.6%) 23 (22.1%) 62 (59.6%) 9 (8.7%) 
LDC was effective in 
improving students’ content 11 (10.6%) 16 (15.4%) 67 (64.4%) 10 (9.6%) 


knowledge. (N = 104) 


LDC modules effectively 
supported students in 
completing writing 
assignments. (N = 104) 


8 (7.7%) 


15 (14.4%) 


67 (64.4%) 


14 (13.5%) 


LDC was effective in 
improving the quality of 
students’ writing. (N = 104) 


9 (8.7%) 


20 (19.2%) 


62 (59.6%) 


13 (12.5%) 


LDC supported students’ 
development of skills needed 
for college and career 
readiness. (N = 104) 


8 (7.7%) 


15 (14.4%) 


69 (66.3%) 


12 (11.5%) 


LDC increased students’ 
capacity to analyze and 
understand the components 
of a writing assignment task. 
(N = 104) 


9 (8.7%) 


15 (14.4%) 


68 (65.4%) 


12 (11.5%) 


LDC helped students develop 
speaking and listening skills. 
(N = 104) 


10 (9.6%) 


17 (16.3%) 


62 (59.6%) 


15 (14.4%) 


Overall, LDC helped improve 
students’ literacy 
performance (N = 104) 


9 (8.7%) 


15 (14.4%) 


66 (63.5%) 


14 (13.5%) 
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Project Liaison Support 


The following question refers to the LDC project liaison in your school. This is the teacher or 
administrator leading your Professional Learning Community work. 


T32. Please indicate the degree to which you agree or disagree with the following 


statements. 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


Our school’s LDC project 
liaison effectively 
supported our Professional 
Learning Community 
meetings. 

(N = 104) 


4 (3.8%) 


13 (12.5%) 


48 (46.2%) 


39 (37.5%) 


When I had questions 
about LDC, I felt 
comfortable approaching 
our school’s project liaison. 
(N = 104) 


4 (3.8%) 


6 (5.8%) 


51 (49%) 


43 (41.3%) 


Our project liaison had a 
strong grasp of LDC. 
(N = 104) 


4 (3.8%) 


12 (11.5%) 


58 (55.8%) 


30 (28.8%) 


Our project liaison helped 
teachers align LDC to 
broader school instructional 
goals. (N = 104) 


4 (3.8%) 


18 (17.3%) 


52 (50.0%) 


30 (28.8%) 


Our project liaison offered 
useful feedback for the 
design and revision of LDC 
modules. (N = 104) 


4 (3.8%) 


20 (19.2%) 


47 (45.2%) 


33 (31.7%) 


Our project liaison was 
effective in inviting 
teachers to join the LDC 
initiative. (N = 104) 


5 (4.8%) 


18 (17.3%) 


49 (47.1%) 


32 (30.8%) 
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School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your 
school. 


T33. What proportion of PLC meetings focused on LDC did your school 
administrator attend? 


(N = 116) 
Less than one quarter of LDC PLCs 32 teachers (27.6%) 
About one quarter of LDC PLCs 11 teachers (9.5%) 
About one half of LDC PLCs 20 teachers (17.2%) 


About three quarters of LDC PLCs 12 teachers (10.3%) 


a than three quarters of LDC 41 teachers (35.3%) 


T34. How many times did your school administrator observe you teach an LDC 
mini-task during the current school year (2016-17)? 


(N = 116) 
0 times 67 teachers (57.8%) 
1 time 29 teachers (25.0%) 
2 times 11 teachers (9.5%) 
30rmoretimes 9 teachers (7.8%) 
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T35. Please indicate the degree to which you agree or disagree with the following 


statements. 
My school administrator... soy Disagree Agree Seon ely 
isagree agree 
had a firm understandin 
of LDC. (N = 115) g 9 (7.8%) 23 (20%) 70 (60.9%) | 13 (11.3%) 
protected common 
planning time for our LDC 
Professional Learning 9 (7.8%) 15 (13.0%) 59 (51.3%) | 32 (27.8%) 
Community to meet. 
(N = 115) 
encouraged teachers to 
participate in LDC. 4 (3.5%) 3 (2.6%) 75 (65.2%) | 33° (28.7%) 


(N = 115) 


expressed concerns that 
implementing LDC is 
taking time away from 
other instructional 
priorities. (N = 115) 


20 (17.4%) 


48 (41.7%) 


36 (31.3%) 


11 (9.6%) 


communicated how using 
LDC’s tools supported 

specific school initiatives 
and/or goals. (N = 115) 


7 (6.1%) 


22 (19.1%) 


66 (57.4%) 


20 (17.4%) 


provided me with 
feedback about my LDC 
planning and/or 
instruction. (N = 115) 


13 (11.3%) 


48 (41.7%) 


46 (40.0%) 


8 (7.0%) 


made formative 
assessment a priority at 
my school. (N = 115) 


8 (7%) 


28 (24.3%) 


63 (54.8%) 


16 (13.9%) 
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Teacher Leadership Role 


T36. Please indicate the degree to which you agree or disagree with the following 
statements about your role in your school’s LDC implementation. 


Strongly Disagree Agree Strongly 
disagree agree 

I was involved in setting 

instructional goals for the LDC 8 (7.8%) | 40 (38.8%) | 41 (39.8%) | 14 (13.6%) 


work at my school. (N = 103) 


I was involved in discussions 
about how to expand LDC 
implementation at my school 
in future years. (N = 103) 


11 (10.7%) 


43 (41.7%) 


40 (38.8%) 


9 (8.7%) 


I had the opportunity to work 
with our LDC project liaison 
and our administrator to help 
shape LDC implementation. 
(N = 103) 


7 (6.8%) 


40 (38.8%) 


47 (45.6%) 


9 (8.7%) 


LDC helped me create writing 
assignments to use within my 
current curriculum. (N = 103) 


7 (6.8%) 


16 (15.5%) 


64 (62.1%) 


16 (15.5%) 


I am interested in learning 
more about how to lead LDC 
implementation at my school 
by facilitating with the virtual 
coach, providing feedback to 
my peers, etc. (N = 103) 


12 (11.7%) 


30 (29.1%) 


48 (46.6%) 


13 (12.6%) 
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Facilitators and Barriers 


statements. 


T37. Please indicate the degree to which you agree or disagree with the following 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


My LDC PLC was given 
sufficient time to meet. 
(N = 115) 


9 (7.8%) 


25 (21.8%) 


6a (56.570) 


16 (13.9%) 


I felt adequately prepared to 
effectively implement LDC 
modules in my classroom. 
(N = 115) 


7 (6.1%) 


39 (33.9%) 


59 (51.3%) 


10 (8.7%) 


It was challenging to find 
content-rich reading 
materials for the LDC 
modules I developed. 

(N = 115) 


7 (6.1%) 


49 (42.6%) 


46 (40.0%) 


13 (11.3%) 


My school had adequate 
technology to support 
teachers’ use of LDC. 

(N = 115) 


9 (7.8%) 


17 (14.8%) 


62 (53.9%) 


27 (23.5%) 


It was easy to find and adapt 
LDC mini-tasks for use in my 
classroom. 
(N = 115) 


9 (7.8%) 


31 (27.0%) 


63 (54.8%) 


12 (10.4%) 
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Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 
e CoreTools online platform 
e LDC online courses in the “Learn” section of CoreTools 
e Virtual coaching 


o Zoom meetings, written feedback on teacher work in LDC CoreTools, Teaching 
Channel discussions, emails, etc. 


e In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


T38. What supports did you find the most useful and why? 


T39. What supports were not helpful and why? 


T40. In what ways could LDC implementation be improved in your school in the 
future? 


Appendix C: 2016-2017 Project Liaison 
Survey and Responses 


LDC Participation 


PL1. Prior to the current school year (2016-17), did you have any experience with 


LDC? 
(N = 18) 


Yes 2 liaisons (11.1%) 
No @Y Skip toPl3 16 liaisons (88.9%) 


PL2. How many of the following did you teach prior to the current school year 
(2016-17)? 
[ ] LDC modules N = 2, Mean = 2, Range: 2-2 


[ ] LDC mini-tasks, outside of modules N = 2, Mean = O, Range: 0-0 


Professional Learning Community and Teacher 
Collaboration 


The following questions involve the LDC Professional Learning Community (PLC) that you are 
leading. 


PL3. About how often did your LDC PLC meet? 


(N = 18) 
Less than once a month 1 liaisons (5.6%) 
Once a month 2 liaisons (11.1%) 
Every other week 8 liaisons (44.4%) 
Once a week BY Skip to PLS 6 liaisons (33.3%) 
Twice a week or more often y Skip to PL5 1 liaisons (5.6%) 
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PL4. What were the primary barriers preventing your LDC PLC from meeting 
weekly? Select all that apply. 


(N = 18) 
PLC time was not protected. 6 liaisons (33.3%) 


PLC members had limited interest in attending 2 liaisons (11.1%) 


meetings. 

School administrator did not make it a priority. 3 liaisons (16.7%) 
I was unable to provide sufficient 1 liaisons (5.6%) 
organizational support. 

Not enough teachers participated. 1 liaisons (5.6%) 
PLC members had other priorities that 7 liaisons (38.9%) 
competed with LDC participation. 

Other (please specify) 1 liaisons (5.6%) 


PL5. About how often did you have informal discussions (as opposed to scheduled 


meetings) about LDC with teachers in your LDC PLC? 
(N = 18) 


Less than once a month 3 liaisons (16.7%) 
Once a month 2 liaisons (11.1%) 
Every other week 8 liaisons (44.4%) 
Once a week 4 liaisons (22.2%) 
Twice a week or more 1 liaisons (5.6%) 


PL6. On average how long did your school’s LDC PLC meetings typically last? 


(N = 18) 
, 2 liaisons 
Less than 45 minutes (11.1%) 
; 15 liaisons 
45 to 59 minutes (83.3%) 
60 to 74 minutes 1 liaisons (5.6%) 
75 minutes or more 0 liaisons (0%) 
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LDC Training and Support 


PL7. How effective was your LDC PLC in the following areas? 


Not 
effective 


A little 
effective 


Moderately 


effective 


Very 
effective 


Demonstrating the 
usefulness of LDC for 
teachers (N = 18) 


1 (5.6%) 


4 (22.2%) 


8 (44.4%) 


5 (27.8%) 


Creating an environment in 
which teachers are 
comfortable working 
together (N = 18) 


1 (5.6%) 


4 (22.2%) 


6 (33.3%) 


7 (38.9%) 


Fostering an environment 
where teachers share their 
instructional plans with 
colleagues (N = 18) 


1 (5.6%) 


4 (22.2%) 


4 (22.2%) 


9 (50.0%) 


Allowing space for shared 
problem solving 
(N = 18) 


2 (11.1%) 


3 (16.7%) 


5 (27.8%) 


8 (44.4%) 


Helping teachers learn to 
develop modules 
(N = 18) 


2 (11.1%) 


2 (11.1%) 


5 (27.8%) 


9 (50.0%) 
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PL8. How would you rate each of the following aspects of the online course 
material (in the Learn tab in LDC CoreTools) that your coach used or directed 


you to use? 


Poor 


Fair 


Good 


Excellent 


Clarity of information 
presented (N = 18) 


0 (0%) 


3 (16.7%) 


11 (61.1%) 


4 (22.2%) 


Relevance of information 
presented (N = 18) 


1 (5.6%) 


2 (11.1%) 


10 (55.6%) 


5 (27.8%) 


Ease of use (N = 18) 


1 (5.6%) 


3 (16.7%) 


11 (61.1%) 


3 (16.7%) 


Usefulness of resource 
documents (e.g., LDC 
Pitfall Checklist, CCSS 
Mental Markers, etc.) 

(N = 18) 


0 (0%) 


4 (22.2%) 


8 (44.4%) 


6 (33.3%) 


Usefulness of videos 
(N = 18) 


1 (5.6%) 


6 (33.3%) 


7 (38.9%) 


4 (22.2%) 


Degree to which course 
material helped teachers 
to create and/or adapt 
LDC modules (N = 18) 


0 (12%) 


6 (33.3%) 


7 (38.9%) 


5 (27.8%) 


Opportunity to extend 
learning when needed 
or desired (N = 18) 


1 (5.6%) 


3 (16.7%) 


5 (27.8%) 


9 (50.0%) 


PL9a. Overall, were you able to get the feedback and support you needed from 
your LDC coach (through written feedback in LDC CoreTools, or coaching and 
modeling in your LDC PLCs) to plan, teach, reflect on, and revise LDC 


modules? 
(N = 18) 
Ved 18 liaisons 
(100%) 


No 0 liaisons (0%) 


PL9b. Did your LDC coach provide written feedback on your module(s) in LDC 
CoreTools in a timely manner? 


(N = 18) 
Yes 17 liaisons (94.4%) 
No 1 liaisons (5.6%) 
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PL10. Outside of the PLC meetings with your LDC coach, please indicate whether 
you used each of the following types of coach support, and how helpful you 


found these types of support. 


Did not 
use 


Used 


Not 
helpful 


A little 
helpful 


Moderately 
helpful 


Very 
helpful 


Written feedback in 
LDC CoreTools from 
your LDC coach (in the 
comments 

areas and/or via 

the teacher 

work rubric) (N = 18) 


4 (22.2%) 


0 (0%) 


2 (11.1%) 


3 (16.7%) 


9 (50.0%) 


One-on-one Zoom 
video conference 
and/or call with your 
LDC coach (N = 18) 


1 (5.6%) 


1 (5.6%) 


1 (5.6%) 


2 (11.1%) 


13 (72.2%) 


Coach-facilitated 
discussions on the 
Teaching Channel 
Teams platform 
(N = 18) 


9 (50.0%) 


1 (5.6%) 


2 (11.1%) 


1 (5.6%) 


5 (27.8%) 


Email or phone 
communication with 
your LDC coach 

(N = 18) 


1 (5.6%) 


0 (0%) 


3 (16.7%) 


1 (5.6%) 


13 (72.2%) 


Other (please specify) 
(N = 2) Solo video 
conference (n=1), 
Texting (n=1) 


0 (0%) 


0 (0%) 


0 (0%) 


0 (0%) 


2 (100%) 


PL11. How many in-person and/or online LDC professional development offerings 
for school administrators and project liaisons did you attend during the 
current school year (e.g., Launch Day [in person], spring Project Liaison 
Meeting [in person], Bi-Monthly Community Meetings [online], Bi-Monthly 
Coach Meetings [online])? 


[] Professional development offerings N = 17, Mean = 3.41, Range: 0-20 
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following statements. 


PL12. Please indicate the degree to which you agree or disagree with the 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


N/A 


I was able to reach my 
LDC coach if I had any 
questions about LDC. 
(N = 18) 


0 (0%) 


0 (0%) 


5 (27.8%) 


13 (72.2%) 


0 (0%) 


LDC provided adequate 
technical support for 
issues with the 
CoreTools online 
platform. (N = 18) 


0 (0%) 


1 (5.6%) 


7 (38.9%) 


9 (50.0%) 


1 (5.6%) 


LDC offered sufficient 
professional 
development 
opportunities for me to 
lead the initiative in 
my school. (N = 18) 


1 (5.6%) 


2 (11.1%) 


4 (22.2%) 


10 (55.6%) 


1 (5.6%) 


LDC coaches were able 
to connect me with 
additional resources 
when needed. 

(N = 18) 


1 (5.6%) 


1 (5.6%) 


6 (33.3%) 


10 (55.6%) 


0 (0%) 


It was challenging to 
coordinate with our 
LDC coach on how to 
structure Professional 
Learning Community 
time. (N = 18) 


3 (16.7%) 


8 (44.4%) 


2 (11.1%) 


4 (22.2%) 


1 (5.6%) 


When I reached out to 
our LDC coach, he or 
she responded quickly. 
(N = 18) 


0 (0%) 


0 (0%) 


3 (16.7%) 


14 (77.8%) 


1 (5.6%) 


Our LDC coach was 
easy to work with. 
(N = 18) 


0 (0%) 


1 (5.6%) 


1 (5.6%) 


15 (83.3%) 


1 (5.6%) 


Our LDC coach was 
knowledgeable and 
provided high quality 
guidance. 

(N = 18) 


0 (0%) 


0 (0%) 


5 (27.8%) 


12 (66.7%) 


1 (5.6%) 
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Module Creation 


PL13. During the current school year (2016-17), how many LDC modules did your 
PLC individually or collaboratively adapt from existing modules (e.g., 
modules created in a prior year and/or modules from the LDC Library in 
CoreTools)? 


[ ] Adapted modules N = 17, Mean = 1.35, Range: 0-2 


PL14. During the current school year (2016-17), how many LDC modules did your 
PLC create (either individually or in a group)? Only include modules built 
from scratch, not those adapted from existing modules in the LDC library. 


[| New modules N = 18, Mean = 1.89, Range: 0-11 


PL15.How did members of your PLC collaborate to create LDC modules? 
Check all that apply. 
(N = 18) 


Modules were created by individual teachers. 5 liaisons (27.8%) 


Modules were created by teams of two or more teachers. 7 liaisons (38.9%) 


Modules were created by the PLC as a whole. 4 liaisons (22.2%) 


Other (please specify) 


3 liaisons (16.7%) 
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when creating LDC modules and/or mini-tasks. 


PL16. Please indicate to what extent you were able to do each of the following 


Not at all 


A little bit 


Toa 
moderate 
extent 


To a great 
extent 


Select a set of focus 
standards for a writing 
assignment. (N = 18) 


0 (0%) 


0 (0%) 


8 (44.4%) 


10 (55.6%) 


Create a standards-driven 
writing assignment task. 
(N = 18) 


0 (0%) 


0 (0%) 


10 (55.6%) 


8 (44.4%) 


Select high quality, 
complex texts and other 
materials that allowed 
students to engage in 
deeper learning. (N = 18) 


0 (0%) 


3 (16.7%) 


8 (44.4%) 


7 (38.9%) 


Create a writing 
assignment that provided 
multiple opportunities for 
students to engage with 
the material. (N =18) 


0 (0%) 


1 (5.6%) 


8 (44.4%) 


9 (50.0%) 


Identify the skills students 
need to develop in order to 
complete a writing 
assignment. (N = 18) 


0 (0%) 


2 (11.1%) 


10 (55.6%) 


6 (33.3%) 


Create daily lessons to 
teach the skills a student 
needs to complete a writing 
assignment. (N = 18) 


0 (0%) 


3 (16.7%) 


9 (50.0%) 


6 (33.3%) 


Plan for a variety of 
methods to assess student 
progress (e.g. mini-task 
scoring guides). (N = 18) 


2 (11.1%) 


5 (27.8%) 


7 (38.9%) 


4 (22.2%) 


Make connections to 
previous or future learning 
that make a writing 
assignment relevant for 
students. (N = 18) 


0 (0%) 


4 (22.2%) 


7 (38.9%) 


7 (38.9%) 


School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your 


school. 


PL17. What proportion of PLC meetings focused on LDC did your school 


administrator attend? 


Less than one quarter of LDC PLCs 
About one quarter of LDC PLCs 
About one half of LDC PLCs 

About three quarters of LDC PLCs 


More than three quarters of LDC 
PLCs 


(N = 16) 

6 liaisons (37.5%) 
4 liaisons (25.0%) 
1 liaisons (6.3%) 
2 liaisons (12.5%) 
3 liaisons (18.8%) 
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PL18. Please indicate the degree to which you agree or disagree with the following 


statements. 


My school administrator... 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


had a firm understanding of 
LDC. (N = 16) 


0 (0%) 


3 (18.8%) 


13 (81.3%) 


0 (0%) 


protected common planning 
time for our LDC 
Professional Learning 
Community to meet. 

(N = 16) 


0 (0%) 


5 (31.3%) 


6 (37.5%) 


5 (31.3%) 


encouraged teachers to 
participate in LDC. 
(N = 16) 


0 (0%) 


0 (0%) 


12 (75%) 


4 (25.0%) 


expressed concerns that 
implementing LDC was 
taking time away from other 
instructional priorities. 

(N = 16) 


5 (31.3%) 


4 (25.0%) 


5 (31.3%) 


2 (12.5%) 


communicated how using 
LDC’s tools supported 
specific school initiatives 
and/or goals. (N = 16) 


0 (0%) 


6 (37.5%) 


7 (43.8%) 


3 (18.8%) 


provided me with feedback 
about my LDC planning 
and/or instruction. 

(N = 16) 


1 (6.3%) 


9 (56.3%) 


5 (31.3%) 


1 (6.3%) 


made formative assessment 
a priority at my school. 
(N = 16) 


1 (6.3%) 


4 (25.0%) 


9 (56.3%) 


2 (12.5%) 
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Project Liaison Leadership Role 


PL19. Please indicate the degree to which you agree or disagree with the following 
statements about your role in leading your school’s LDC implementation. 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


I met regularly with my 
school administrator to 
make planning decisions 
around LDC. (N = 16) 


0 (0%) 


9 (56.3%) 


5 (31.3%) 


2 (12.5%) 


I was involved in 
discussions about 
differentiating LDC 
implementation to meet 
teacher learning needs. 
(N = 16) 


0 (0%) 


5 (31.3%) 


6 (37.5%) 


5 (31.3%) 


I was involved in 
discussions about how to 
expand LDC 
implementation at my 
school in future years. 
(N = 16) 


0 (0%) 


7 (43.8%) 


4 (25.0%) 


5 (31.3%) 


My role as a LDC project 
liaison allowed me to 
effectively advocate for 
additional resources on 
my campus. (N = 16) 


0 (0%) 


7 (43.8%) 


4 (25.0%) 


5 (31.3%) 


I was involved in setting 
instructional goals for the 
LDC work at my school. 
(N = 16) 


0 (0%) 


5 (31.3%) 


5 (31.3%) 


6 (37.5%) 


I met regularly with my 
LDC coach to manage the 
LDC work plan. (N = 16) 


0 (0%) 


2 (12.5%) 


10 (62.5%) 


4 (25.0%) 


I feel that my position as 
an LDC project liaison 
allowed me to build my 
capacity as an 
instructional leader 
among my colleagues. 
(N = 16) 


0 (0%) 


4 (25.0%) 


4 (25.0%) 


8 (50.0%) 
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7 


Alignment 


statements. 


PL20. Please indicate the degree to which you agree or disagree with the following 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


Our school connected LDC 
implementation to our 
specific school wide goals. 
(N = 18) 


0 (0%) 


8 (44.4%) 


6 (33.3%) 


4 (22.2%) 


LDC helped teachers create 
writing assignments to use 
within their current 
curricula. (N = 18) 


0 (0%) 


4 (22.2%) 


11 (61.1%) 


3 (16.7%) 


LDC complemented other 
initiatives taking place in my 
school. (N = 18) 


1 (5.6%) 


2 (11.1%) 


11 (61.1%) 


4 (22.2%) 


LDC was a strategy for 
implementing my state’s 
College- and Career-Ready 
Standards. (N = 18) 


0 (0%) 


3 (16.7%) 


12 (66.7%) 


3 (16.7%) 


The time spent 
implementing LDC 
interfered with other 
important initiatives at my 
school. (N = 18) 


3 (16.7%) 


6 (33.3%) 


7 (38.9%) 


2 (11.1%) 


LDC helped prepare 
students in my school for 
current state assessments. 
(N = 18) 


0 (0%) 


5 (27.8%) 


10 (55.6%) 


3 (16.7%) 


It was difficult for teachers 
to focus on LDC because of 
other competing priorities at 
the school. (N = 18) 


1 (5.6%) 


4 (22.2%) 


5 (27.8%) 


8 (44.4%) 
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Scale-up and Sustainability 


statements. 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


I expect that most 
teachers participating in 
LDC this year will 
continue to do so next 
year. (N = 18) 


1 (5.6%) 


4 (22.2%) 


12 (66.7%) 


1 (5.6%) 


Teachers at my school 
who were not part of 
the LDC PLC meetings 
used the LDC planning 
process and/or LDC 
CoreTools. (N = 18) 


6 (33.3%) 


7 (38.9%) 


5 (27.8%) 


0 (0%) 


As a result of LDC, new 
collaborations across 
grades and/or subjects 
were created or are 
being launched at my 
school. (N = 18) 


3 (16.7%) 


9 (50.0%) 


6 (33.3%) 


0 (0%) 


Teachers and 
administrators at my 
school are committed to 
sustaining the LDC 
initiative. (N = 18) 


1 (5.6%) 


4 (22.2%) 


11 (61.1%) 


2 (11.1%) 


PL21. Please indicate the degree to which you agree or disagree with the following 


I expect our LDC PLC to 
increase in size next 
year. (N = 18) 


1 (5.6%) 


5 (27.8%) 


9 (50.0%) 


3 (16.7%) 
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District Support 


following statements. 


PL22. Please indicate the degree to which you agree or disagree with the 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


Don’t 
know 


District leaders 
supported the 
implementation of 
LDC. (N = 18) 


0 (0%) 


0 (0%) 


6 (33.3%) 


9 (50.0%) 


3 (16.7%) 


District leaders had 
a firm 
understanding 

of LDC. (N = 18) 


0 (0%) 


1 (5.6%) 


6 (33.3%) 


6 (33.3%) 


5 (27.8%) 


District leaders are 
interested in 
spreading the use of 
LDC to additional 
schools. (N = 18) 


0 (0%) 


1 (5.6%) 


6 (33.3%) 


7 (38.9%) 


4 (22.2%) 


District professional 
development efforts 
were aligned with 
the LDC initiative. 
(N = 18) 


0 (0%) 


2 (11.1%) 


6 (33.3%) 


4 (22.2%) 


6 (33.3%) 


District leaders 
visited my school to 
discuss the 
implementation of 
LDC. (N = 18) 


1 (5.6%) 


1 (5.6%) 


8 (44.4%) 


5 (27.8%) 


3 (16.7%) 
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Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 
e CoreTools online platform 
e LDC online courses in the “Learn” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, Teaching 
Channel discussions, emails, etc. 
e In-person coaching 
o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


PL23. What supports did you find the most useful and why? 


PL24. What supports were not helpful and why? 


PL25. In what ways could LDC implementation be improved in your school in the 
future? 
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Appendix D: 2016-2017 Administrator 
Survey and Responses 


LDC Participation 


A1. What is your role at the school? 


(N = 21) 
Principal 14 admins (66.7%) 
Assistant Principal 7 admins (33.3%) 
Other (please specify) 0 admins (0%) 


Professional Learning Community 


A2. What proportion of LDC Professional Learning Community (PLC) meetings did 
you attend during the current school year? 


(N = 20) 
Less than one quarter of LDC PLCs 7 admins (35%) 


About one quarter of LDC PLCs 3 admins (15%) 
About one half of LDC PLCs 4 admins (20%) 
About three quarters of LDC PLCs 2 admins (10%) 


More than three quarters of LDC 4 admins (20%) 
PLCs 
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Training and Support 


A3. How many in-person and/or online LDC professional development offerings for 
school administrators and project liaisons did you attend during the current 
school year (e.g., Launch Day [in person], February Administrator Meeting [in 
person], Bi-Monthly Community Meetings [online])? 


LDC Professional 


development offerings N = 20, Mean = 1.85, Range: 0-5 


A4. Please indicate the degree to which you agree or disagree with the following 
statements. 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


N/A 


I was able to reach 
LDC staff when I had 
questions about LDC. 
(N = 20) 


1 (5.0%) 


0 (0%) 


7 (35.0%) 


12 (60.0%) 


0 (0%) 


My school has 

adequate technology 
to access LDC online 
resources. (N = 20) 


2 (10.0%) 


0 (0%) 


10 (50.0%) 


8 (40.0%) 


0 (0%) 


LDC offered sufficient 
professional 
development 
opportunities for LDC 
project liaisons. 

(N = 20) 


1 (5.0%) 


1 (5.0%) 


11 (55.0%) 


5 (25.0%) 


2 (10.0%) 


LDC offered sufficient 
professional 
development 
opportunities for 
school 
administrators. 

(N = 20) 


1 (5.0%) 


3 (15.0%) 


13 (65.0%) 


2 (10.0%) 


1 (5.0%) 


LDC staff members 
were able to connect 
me with additional 
resources when 
needed. 

(N = 20) 


1 (5.0%) 


0 (0%) 


12 (60.0%) 


5 (25.0%) 


2 (10.0%) 
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Classroom Observation 


A5. How many times during the current school year did you observe teachers 
implementing an LDC module? 


(N = 20) 
0 times @Y Skip toA7 1 admins (5.0%) 
1-2 times 8 admins (40.0%) 
3-5 times 9 admins (45.0%) 
6 or moretimes 2 admins (10.0%) 


A6. On average, how effective were LDC modules in developing students’ literacy 


skills? 
(N = 19) 
Not effective 0 admins (0%) 
A little effective 3 admins (15.8%) 
Moderately effective 12 admins (63.2%) 
Very effective 4 admins (21.1%) 
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Impact on Teacher Practice 


A7. Based on your oversight of the LDC program, please indicate on average how 


much the teaching practice of LDC PLC members improved in each of the 


following areas: 


Not at all 


A little 


Moderately 


A great 
deal 


Selecting a set of focus 
standards for a writing 
assignment (N = 20) 


0 (0%) 


3 (15.0%) 


9 (45.0%) 


8 (40.0%) 


Creating standards-driven 
writing tasks of sufficient 
quality and scope to allow 
students to think deeply 
about disciplinary content 
and practice literacy skills 
(N = 20) 


0 (0%) 


3 (15.0%) 


10 (50.0%) 


7 (35.0%) 


Identifying the skills 
students need to develop in 
order to complete a writing 
assignment (N = 20) 


0 (0%) 


3 (15.0%) 


10 (50.0%) 


7 (35.0%) 


Creating daily lessons to 
teach each skill a student 
needs to complete a writing 
assignment (N = 20) 


0 (0%) 


6 (30.0%) 


10 (50.0%) 


4 (20.0%) 


Assessing students’ 
progress as they work 
toward completing a writing 
assignment (N = 20) 


0 (0%) 


6 (30.0%) 


10 (50.0%) 


4 (20.0%) 


Tracking and analyzing 
evidence about student 
progress in a systematic 
way (N = 20) 


0 (0%) 


6 (30.0%) 


10 (50.0%) 


4 (20.0%) 


Identifying patterns of 
student understandings or 
misconceptions (N = 20) 


2 (10%) 


4 (20.0%) 


13 (65.0%) 


1 (5.0%) 


Using evidence of student 
performance of standards to 
shape future instructional 
decisions (N = 20) 


0 (0%) 


5 (25.0%) 


10 (50.0%) 


5 (25.0%) 
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Impact on Student Learning 


below: 


A8. Please indicate the degree to which you agree or disagree with the statements 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


LDC helped students 


develop reading skills. 


(N = 20) 


0 (0%) 


4 (20.0%) 


13 (65.0%) 


3 (15.0%) 


LDC was effective in 
improving students’ 
content knowledge. 
(N = 20) 


0 (0%) 


3 (15.0%) 


14 (70.0%) 


3 (15.0%) 


LDC modules 
effectively supported 
students in 
completing writing 
assignments. 

(N = 20) 


0 (0%) 


1 (5.0%) 


13 (65.0%) 


6 (30.0%) 


LDC was effective in 
improving the quality 
of students’ writing. 
(N = 20) 


0 (0%) 


2 (10.0%) 


14 (70.0%) 


4 (20.0%) 


LDC supported 
students’ 
development of skills 
needed for college 
and career readiness. 
(N = 20) 


0 (0%) 


1 (5.0%) 


13 (65.0%) 


6 (30.0%) 


LDC increased 
students’ capacity to 
analyze and 
understand the 
components of a 
writing assignment 
task. (N = 20) 


0 (0%) 


3 (15.0%) 


13 (65.0%) 


4 (20.0%) 


LDC helped students 
develop speaking and 
listening skills. 

(N = 20) 


0 (0%) 


4 (20.0%) 


13 (65.0%) 


3 (15.0%) 


Overall, LDC helped 
improve students’ 
literacy performance. 
(N = 20) 


0 (0%) 


1 (5.0%) 


13 (65.0%) 


6 (30.0%) 
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y¥4) Administrator Leadership Role 


AQ. Please indicate the degree to which you agree or disagree with the following 
statements about your role in leading LDC implementation in your school: 


strongly Disagree Agree Strongly, 
disagree agree 


I was able to shape LDC 
implementation at my 0 (0%) 4 (20.0%) 12 (60.0%) 4 (20.0%) 
school. (N = 20) 


I met regularly with the 
LDC project liaison in my 
school to stay abreast of 0 (0%) 5 (27.8%) 11 (61.1%) | 2 (11.1%) 
implementation progress. 
(N = 18) 


I was involved in 
discussions about 
differentiating LDC 
implementation to meet 
teacher learning needs. 
(N = 20) 


0 (0%) 6 (30.0%) | 12 (60.0%) | 2 (10.0%) 


I led discussions about how 
to expand my school’s LDC 
implementation in future 
years. (N =20) 


1 (5.0%) 7 (35.0%) | 10 (50.0%) | 2 (10.0%) 


I made changes to school 
schedules to accommodate 
LDC professional learning 
time. (N = 20) 


0 (0%) 315.0%) | 12 (60.0%) | 5 (25.0%) 
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Alignment 


statements: 


A10. Please indicate the degree to which you agree or disagree with the following 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


Our school connected LDC 
implementation to our 
specific school wide goals. 
(N = 20) 


0 (0%) 


7 (35.0%) 


10 (50.0%) 


3 (15.0%) 


LDC helped teachers create 
writing assignments to use 
within their current 
curricula. (N = 20) 


0 (0%) 


3 (15.0%) 


10 (50.0%) 


7 (35.0%) 


LDC complemented other 
initiatives taking place in 
my school. (N = 20) 


0 (0%) 


4 (20.0%) 


9 (45.0%) 


7 (35.0%) 


LDC was a strategy for 
implementing my state’s 
College- and Career-Ready 
Standards. (N = 20) 


0 (0%) 


3 (15.0%) 


13 (65.0%) 


4 (20.0%) 


The time spent 
implementing LDC 
interfered with other 
important initiatives at my 
school. (N = 20) 


6 (30.0%) 


7 (35.0%) 


7 (35.0%) 


0 (0%) 


LDC helped prepare 
students in my school for 
current state assessments. 
(N = 20) 


0 (0%) 


7 (35.0%) 


12 (60.0%) 


1 (5.0%) 


It was difficult for teachers 
to focus on LDC because of 
other competing priorities 
at the school. (N = 20) 


1 (5.0%) 


5 (25.0%) 


12 (60.0%) 


2 (10.0%) 
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Scale-up and Sustainability 


statements: 


A11. Please indicate the degree to which you agree or disagree with the following 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


Don’t 
know 


I expect that most 
teachers 
participating in LDC 
this year will 
continue to do so 
next year. (N = 20) 


0 (0%) 


5 (25.0%) 


7 (35.0%) 


7 (35.0%) 


1 (5.0%) 


Teachers at my 
school who were not 
part of the LDC PLC 
meetings used the 
LDC planning process 
and/or LDC 
CoreTools. (N = 20) 


4 (20.0%) 


12 (60.0%) 


3 (15.0%) 


0 (0%) 


1 (5.0%) 


As a result of LDC, 
new collaborations 
across grades and/or 
subjects were 
created or are being 
launched at my 
school. (N = 20) 


2 (10.0%) 


10 (50.0%) 


5 (25.0%) 


3 (15.0%) 


0 (0%) 


Teachers and 
administrators at my 
school are committed 
to sustaining the LDC 
initiative. (N = 20) 


1 (5.0%) 


3 (15.0%) 


8 (40.0%) 


6 (30.0%) 


2 (10.0%) 


I expect our LDC PLC 
to increase in size 
next year. (N = 20) 


1 (5.0%) 


6 (30.0%) 


6 (30.0%) 


5 (25.0%) 


2 (10.0%) 
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District Support 


statements: 


A12. Please indicate the degree to which you agree or disagree with the following 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


Don’t 
know 


District leaders 
supported the 
implementation of 
LDC. (N = 20) 


0 (0%) 


4 (20.0%) 


9 (45.0%) 


4 (20.0%) 


3 (15.0%) 


District leaders had a 
firm understanding 
of LDC. (N = 20) 


0 (0%) 


7 (35.0%) 


4 (20.0%) 


3 (15.0%) 


6 (30.0%) 


District leaders are 
interested in 
spreading the use of 
LDC to additional 
schools. 

(N = 20) 


0 (0%) 


3 (15.0%) 


6 (30.0%) 


3 (15.0%) 


8 (40.0%) 


District professional 
development efforts 
were aligned with the 
LDC initiative. 

(N = 20) 


0 (0%) 


7 (35.0%) 


4 (20.0%) 


5 (25.0%) 


4 (20.0%) 


District leaders 
visited my school to 
discuss the 
implementation of 
LDC. (N = 20) 


2 (10.0%) 


5 (25.0%) 


7 (35.0%) 


4 (20.0%) 


2 (10.0%) 
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11 Areas for Improvement 


There have been a number of supports for implementation of LDC in your 
school, including: 
e CoreTools online platform 
e LDC online courses in the “Learn” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, 
Teaching Channel discussions, emails, etc. 
e In-person coaching 
o Summer training, in-person support visits from LDC and District Lead, in- 
person professional development opportunities, etc. 


A13. What supports did you find the most useful and why? 


A14. What supports were not helpful and why? 


A15. In what ways could LDC implementation be improved in your 
school in the future? 
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Appendix F: Outcome Analysis Methodology 


Analysis Model Specification. For our outcome analyses, we used a threshold of p < 
.05 to determine whether there was a statistically significant impact of LDC on ELA 
achievement. In addition to the LDC treatment indicator, the fixed effects included a school 
cohort indicator, a teacher effect for years of experience, as well as an aggregate indicator 
measuring prior performance of each student’s classroom peers. The fixed effects also 
included student characteristics to identify the matched comparison sample of students, such 
as baseline achievement, socio-economic status, demographics, language proficiency, grade 


level, and participation in special education. 


The three-level MMMC model was used to estimate the impacts of the LDC intervention 
on student learning. This same analytic model will be used to estimate impacts in future years. 
The general specification for the middle school level MMMC model is shown in the following 
equation using similar notation proposed by Browne et al. (2001, equation 6) and applied in 
Tranmer et al. (2014, equation 3). 


_ oo (3) (2) 
Y= xX B+ Uschool(i) > Wi ju; +e; 


j€Teacher(i) 


i = 1,..,n Teacher(i) C (1,...,J) 


(3) 2 (2) 
Uschool(i) ~ N(0, ox), uo ~ N(0, o%¢0)); e, ~ N(0,o2) 


In this model yi is the student achievement score response, Xiis a vector of the fixed 


covariates and £ is the vector of the corresponding fixed effects. School(i) is the school which 
studenti attends, thus the term eee represents the random effects for that level of 


classification. Within the term Yjereacner() Wi ju), wy is the set of j random effects for the 


teachers included in the selected data set, and w;; is the weight which sums to 1 for each 
student applied in proportion to the instruction time assigned with each teacher. The following 
presents an example of the full model middle school specification. 


achievement; = By + LDC; * 6, + priorELA; * Bz + priorMath, * B, + Female; * 6, + LEP; 
* Bs + SES; * B, + Black; * B; + Hispanic; * Bg + SPED; * By + Grade7; * B49 
+ Grade8; * Bi, + Gifted; * B,. + priorClassELA; * B,3 + TeachExp; * By4 


(3) 2 
+ TeachAbs; * Bys + Uschoor(i) Wj, ju 2 4. e; 


j€Teacher(i) 


(3) 2 (2) 
Uschool(i) ~ N(0, o%¢3)), uj ~ N(0, o%¢0)); e, ~ N(0,o2) 
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In this model, achievement; is the standardized ELA outcome score for student i; In the 
dosage dependent model LDC; is the proportion of core class instruction time taken with an 
LDC teacher (ranges 0-1); In the dosage independent model LDC, is coded as zero for 


comparison students and as 1 for students receiving any level of LDC teacher exposure. 


Female;, LEP;, Black;, Hispanic; SPED;, Gifted;,and SES; are student demographic indicators 


coded 1 if the status is present and 0 if absent; 


Grade7,; , and Grade8,; are dummy coded 1 when the grade was attended and otherwise 
as O, allowing grade 6 students to serve as the reference group. In the models that combine 
elementary and middle school students the additional dummy coded grade variables will be 
included. The elementary school specification would be very similar, with different dummy 


variables for grades; 


priorELA; , and priorMath, are standardized student achievement scores from the prior 


year; 


priorClassELA; is the aggregated mean of the prior ELA scores for all the core class peers 
of student i; 


TeachExp,; is the aggregated mean of the years of teaching experience for those teachers 


which student i was exposed to in her core classes; 


TeachAbs; is the aggregated mean of the number of days absent for those teachers which 


student i was exposed to in her core classes; 
B, is the impact of LDC, the treatment; 
By2 is the average difference between cohort 1 and cohort 2; 
B, & B; are the effects of the prior score covariates; 
By. By are the effects of the demographic covariates; 


Biz. By, are the effects of the aggregated class level covariates 


(3) 
Uschool(i) U 


respectively assumed to all have mean 0 and variance, 0,2), 0;(3),0¢ respectively. 


u;,e, are the error components at the school, teacher, and student-level 


138 


Table A.F.1 


Elementary School Student/Teacher Weighting Based on Course Mark Links 


Content Area 


Course Mark 


Weighting 


ELA Listening These eight course mark linkages are 
Reading weighted proportionally so that they sum 
Speaking to equal 1/9 in any given grading period 
” and 1/3 across a full year. 
Writing 
Listening - ELD 
Reading - ELD 
Speaking - ELD 
Writing - ELD 
Social studies |Hist Soc Sci This course mark linkage is weighted 
proportionally to sum to equal 1/9 in any 
given grading period and 1/3 across a full 
year. 
Science This course mark linkage is weighted 
Science proportionally to sum to equal 1/9 in any 
given grading period and 1/3 across a full 
year. 
Other Health Ed These course mark linkages are not 
Physical Ed included in the student/teacher weighting 
Mathematics procedure 
Arts 
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Table A.F.2 


Middle School Student/Teacher Weighting Based on Course Mark Links 


nae Example LDC 
Content area | Possible courses EXaiple Slee nee ny Weldheig er treatment 
use in MMMC 
dosage 
Core ELA ENGLISH —Regular |Student takes two units Core ELA with an |Weight=0.25 
and Honors intervention Teacher: 
English 6A & 6B 
6A,6B,7A,7B,8A,8b 
Weight=2/8=0.25 
Supplemental |ELD, ADV ELD, Student takes two supplemental units Weight=0.25 
ELA Literature & ELA with and an LDC teacher: 
Language, Lit Success 1A &1B 
Literature Success 
Weight=2/8=0.25 
Social studies |WHG ANC CIV, Student takes two units of social studies |Weight=0.0 
WHG MED/MOD, with a non-intervention teacher: 
US HIST G&C WHG ANC CIVA & B 
A & B (Regular and 
Honors) : 
Weight=2/8=0.25 
Also HUMANIT A&B 
SCI/HLTH 6A &6B_ |Student takes two units social studies Weight=0.0 
Science HEALTH JH with a non-intervention teacher: 
SCIENCE 7 SCI/HLTH 6A & 6B 
SCIENCE 8A & 8B 
(Regular and Weight=2/8=0.25 
Honors) 
Other All other Courses — |Not included in analysis Not included 
in analysis 
Total Unity: for every student the Treatment 
student/teacher weights sum to 1 Weight=0.50 


Note. The acronyms used by the district are used in the lists of possible courses. 


Calculation of Effect Size 


We calculated student-level effect sizes according to the WWC 3.0 criteria. Specifically, 
for the impact analysis with treatment status as a dichotomous variable, we calculated 
Hedges’ g, the difference in adjusted mean outcomes for the groups divided by the unadjusted 
pooled within-group standard deviation of the outcome measure in the sample, for all 
outcomes. The difference in adjusted mean outcomes is estimated by f£, in the models defined 
above, as outlined in the WWC standards handbook for computing effect sizes in multi-level 
frameworks. Specifically, 
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wy 
CO 
(mj - 1)s? + (n, - 1)83 
njtn, — 2 


where y is £, which is the coefficient from the MMMC for the intervention effect. 


Note that in our analyses the outcome measure is standardized within the analytical 
sample (mean=O, standard deviation=1). As a result, we expected that g would likely be 
quite similar to the ~, coefficient from the MMMC Model in the large samples we plan to collect 
later in the study. 


Though it is not standard to use g with a continuous treatment effect, as in the case of 
our dosage dependent treatment measure, we have defined and matched populations n,; and 
n, where treated students (n;) could have any positive treatment value<=1, and control 
students (n,) had a treatment value of zero. We could therefore calculate g in the case of our 
dosage dependent treatment measure, and again expect that it would not differ substantially 
from the £, coefficient. It was crucial, however, to note that g and £, in the dosage dependent 
models reflect the effect size projected for a student who would receive exposure to 
intervention teachers in all of their core classes. Along with this effect we report average 
dosage received by treated students so that the average treatment effect on treated students 


could be calculated. 
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